Volume leveler controller and controlling method

ABSTRACT

Volume leveler controller and controlling method are disclosed. In one embodiment, A volume leveler controller includes an audio content classifier for identifying the content type of an audio signal in real time; and an adjusting unit for adjusting a volume leveler in a continuous manner based on the content type as identified. The adjusting unit may configured to positively correlate the dynamic gain of the volume leveler with informative content types of the audio signal, and negatively correlate the dynamic gain of the volume leveler with interfering content types of the audio signal.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/509,791, filed Jul. 12, 2019, which is a divisional of U.S. patentapplication Ser. No. 15/432,679 (now U.S. Pat. No. 10,411,669), filedFeb. 14, 2017, which is a divisional of U.S. patent application Ser. No.15/284,953 (now U.S. Pat. No. 9,923,536), filed Oct. 4, 2016, which is adivisional of U.S. patent application Ser. No. 14/777,271 (now U.S. Pat.No. 9,548,713), filed Sep. 15, 2015, which is the 371 national stage ofPCT/US2014/030385, filed Mar. 17, 2014, which claims priority to ChinesePatent Application No. 201310100422.1, filed Mar. 26, 2013 and claimsthe benefit of the U.S. Provisional Patent Application No. 61/811,072,filed Apr. 11, 2013, each of which is hereby incorporated by referencein its entirety.

TECHNICAL FIELD

The present application relates generally to audio signal processing.Specifically, embodiments of the present application relate toapparatuses and methods for audio classifying and processing, especiallycontrolling of dialog enhancer, surround virtualizer, volume leveler andequalizer.

BACKGROUND

Some audio improving devices tend to modify audio signals, in eithertemporal domain or spectral domain, in order to improve overall qualityof the audio and enhance users' experience correspondingly. Variousaudio improving devices have been developed for various purposes. Sometypical examples of audio improving devices include:

Dialog Enhancer: Dialog is the most important component in a movie andradio or TV program to understand the story. Methods were developed toenhance the dialogs in order to increase their clarity and theirintelligibility, especially for elders with decreasing hearingcapability.

Surround Virtualizer: A surround virtualizer enables a surround(multi-channel) sound signal to be rendered over the internal speakersof the PC or over headphones. That is, with the stereo device (such asspeakers and headphones), it creates virtually surround effect andprovides cinematic experience for consumers.

Volume Leveler: A volume leveler aims at tuning the volume of the audiocontent on playback and keeping it almost consistent over the timelinebased on a target loudness value.

Equalizer: An equalizer provides consistency of spectral balance, asknown as “tone” or “timbre”, and allows users to configure the overallprofile (curve or shape) of the frequency response (gain) on eachindividual frequency band, in order to emphasize certain sounds orremove undesired sounds. In a traditional equalizer, different equalizerpresets may be provided for different sounds, such as different musicgenres. Once a preset is selected, or an equalization profile is set,the same equalization gains will be applied on the signal, until theequalization profile is modified manually, In contrast, a dynamicequalizer achieves the spectral balance consistency by continuouslymonitoring the spectral balance of the audio, comparing it to a desiredtone, and dynamically adjusting an equalization filter to transform theaudio's original tone into the desired tone.

In general, an audio improving devices has its own applicationscenario/context. That is, an audio improving device may be suitable foronly a certain set of contents but not for all the possible audiosignals, since different contents may need to be processed in differentways. For example, a dialog enhancement method is usually applied onmovie content. If it is applied on music in which there are no dialogs,it may falsely boost some frequency sub-bands and introduce heavy timbrechange and perceptual inconsistency. Similarly, if a noise suppressionmethod is applied on music signals, strong artifacts will be audible.

However, for an audio processing system that usually comprises a set ofaudio improving devices, its input could be unavoidably all the possibletypes of audio signals. For example, an audio processing system,integrated in a PC, will receive audio content from a variety ofsources, including movie, music, VoIP and game. Thus, identifying ordifferentiating the content being processed becomes important, in orderto apply better algorithms or better parameters of each algorithm on thecorresponding content.

In order to differentiate audio content and apply better parameters orbetter audio improving algorithms correspondingly, traditional systemsusually pre-design a set of presets, and users are asked to choose apreset for the content being played. A preset usually encodes a set ofaudio improving algorithms and/or their best parameters that will beapplied, such as a ‘Movie’ preset and a ‘Music’ preset which isspecifically designed for movie or music playback.

However, manual selection is inconvenient for users. Users usually don'tfrequently switch among the predefined presets but just keep using onepreset for all the content. In addition, even in some automaticsolutions the parameters or algorithms setup in the presets are usuallydiscrete (such as turn On or Off for a specific algorithm with respectto a specific content), it cannot adjust parameters in a content-basedcontinuous manner.

SUMMARY

The first aspect of the present application is to automaticallyconfigure audio improving devices in a continuous manner based on theaudio content on playback. With this “automatic” mode, users can simplyenjoy their content without bothering to select different presets. Onthe other hand, continuously tuning is more important in order to avoidaudible artifacts at the transition points.

According to an embodiment of the first aspect, an audio processingapparatus includes an audio classifier for classifying an audio signalinto at least one audio type in real time; an audio improving device forimproving experience of audience; and an adjusting unit for adjusting atleast one parameter of the audio improving device in a continuous mannerbased on the confidence value of the at least one audio type.

The audio improving device may be any of dialog enhancer, surroundvirtualizer, volume leveler and equalizer.

Correspondingly, an audio processing method includes: classifying anaudio signal into at least one audio type in real time; and adjusting atleast one parameter for audio improvement in a continuous manner basedon the confidence value of the at least one audio type.

According to another embodiment of the first aspect, A volume levelercontroller includes an audio content classifier for identifying thecontent type of an audio signal in real time; and an adjusting unit foradjusting a volume leveler in a continuous manner based on the contenttype as identified. The adjusting unit may configured to positivelycorrelate the dynamic gain of the volume leveler with informativecontent types of the audio signal, and negatively correlate the dynamicgain of the volume leveler with interfering content types of the audiosignal.

Also disclosed is an audio processing apparatus comprising A volumeleveler controller as stated above.

Correspondingly, A volume leveler controlling method includes:identifying the content type of an audio signal in real time; andadjusting a volume leveler in a continuous manner based on the contenttype as identified, by positively correlating the dynamic gain of thevolume leveler with informative content types of the audio signal, andnegatively correlating the dynamic gain of the volume leveler withinterfering content types of the audio signal.

According to yet another embodiment of the first aspect, an equalizercontroller includes an audio classifier for identifying the audio typeof an audio signal in real time; and an adjusting unit for adjusting anequalizer in a continuous manner based on the confidence value of theaudio type as identified.

Also disclosed is an audio processing apparatus comprising an equalizercontroller as stated above.

Correspondingly, an equalizer controlling method includes: identifyingthe audio type of an audio signal in real time; and adjusting anequalizer in a continuous manner based on the confidence value of theaudio type as identified.

The present application also provides a computer-readable medium havingcomputer program instructions recorded thereon, when being executed by aprocessor, the instructions enabling the processor to execute theabove-mentioned audio processing method, or The volume levelercontrolling method, or the equalizer controlling method.

According to the embodiments of the first aspect, the audio improvingdevice, which may be one of dialog enhancer, surround virtualizer,volume leveler and equalizer, may be continuously adjusted according tothe type of the audio signal and/or the confidence value of the type.

The second aspect of the present application is to develop a contentidentification component to identify multiple audio types, and thedetected results may be used to steer/guide the behaviors of variousaudio improving devices, by finding better parameters in a continuousmanner.

According to an embodiment of the second aspect, an audio classifierincludes: a short-term feature extractor for extracting short-termfeatures from short-term audio segments each comprising a sequence ofaudio frames; a short-term classifier for classifying a sequence ofshort-term segments in a long-term audio segment into short-term audiotypes using respective short-term features; a statistics extractor forcalculating the statistics of the results of the short-term classifierwith respect to the sequence of short-term segments in the long-termaudio segment, as long-term features; and a long-term classifier for,using the long-term features, classifying the long-term audio segmentinto long-term audio types.

Also disclosed is an audio processing apparatus comprising an audioclassifier as stated above.

Correspondingly, an audio classifying method includes: extractingshort-term features from short-term audio segments each comprising asequence of audio frames; classifying a sequence of short-term segmentsin a long-term audio segment into short-term audio types usingrespective short-term features; calculating the statistics of theresults of classifying operation with respect to the sequence ofshort-term segments in the long-term audio segment, as long-termfeatures; and classifying the long-term audio segment into long-termaudio types using the long-term features.

According to another embodiment of the second aspect, an audioclassifier includes: an audio content classifier for identifying acontent type of a short-term segment of an audio signal; and an audiocontext classifier for identifying a context type of the short-termsegment at least partly based on the content type identified by theaudio content classifier.

Also disclosed is an audio processing apparatus comprising an audioclassifier as stated above.

Correspondingly, an audio classifying method includes: identifying acontent type of a short-term segment of an audio signal; and identifyinga context type of the short-term segment at least partly based on thecontent type as identified.

The present application also provides a computer-readable medium havingcomputer program instructions recorded thereon, when being executed by aprocessor, the instructions enabling the processor to execute theabove-mentioned audio classifying methods.

According to the embodiments of the second aspect, an audio signal maybe classified into different long-term types or context types, which aredifferent from short-term types or content types. The types of the audiosignal and/or the confidence value of the types may be further used toadjust an audio improving device, such as dialog enhancer, surroundvirtualizer, volume leveler or equalizer.

BRIEF DESCRIPTION OF DRAWINGS

The present application is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings, in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is a diagram illustrating an audio processing apparatus accordingto an embodiment of the application;

FIGS. 2 and 3 are diagrams illustrating variants of the embodiment asshown in FIG. 1;

FIGS. 4-6 are diagrams illustrating possible architecture of classifiersfor identifying multiple audio types and the calculation of confidencevalue;

FIGS. 7-9 are diagrams illustrating more embodiments of the audioprocessing apparatus of the present application;

FIG. 10 is a diagram illustrating delay of transition between differentaudio types;

FIGS. 11-14 are flow charts illustrating an audio processing methodaccording to embodiments of the present application;

FIG. 15 is a diagram illustrating a dialog enhancer controller accordingto an embodiment of the present application;

FIGS. 16 and 17 are flowcharts illustrating the use of the audioprocessing method according to the present application in thecontrolling of a dialog enhancer;

FIG. 18 is a diagram illustrating a surround virtualizer controlleraccording to an embodiment of the present application;

FIG. 19 is a flowchart illustrating the use of the audio processingmethod according to the present application in the controlling of asurround virtualizer;

FIG. 20 is a diagram illustrating a volume leveler controller accordingto an embodiment of the present application;

FIG. 21 is a diagram illustrating the effect of the volume levelercontroller according to the present application;

FIG. 22 is a diagram illustrating an equalizer controller according toan embodiment of the present application;

FIG. 23 illustrates several examples of desired spectral balancepresets;

FIG. 24 is a diagram illustrating an audio classifier according to anembodiment of the present application;

FIGS. 25 and 26 are diagrams illustrating some features to be used bythe audio classifier of the present application;

FIGS. 27-29 are diagrams illustrating more embodiments of the audioclassifier according to the present application;

FIGS. 30-33 are flow charts illustrating an audio classifying methodaccording to embodiments of the present application;

FIG. 34 is a diagram illustrating an audio classifier according toanother embodiment of the present application;

FIG. 35 is a diagram illustrating an audio classifier according to yetanother embodiment of the present application;

FIG. 36 is a diagram illustrating heuristic rules used in the audioclassifier of the present application;

FIGS. 37 and 38 are diagrams illustrating more embodiments of the audioclassifier according to the present application;

FIGS. 39 and 40 are flow charts illustrating an audio classifying methodaccording to embodiments of the present application;

FIG. 41 is a block diagram illustrating an exemplary system forimplementing embodiments of the present application.

DETAILED DESCRIPTION

The embodiments of the present application are below described byreferring to the drawings. It is to be noted that, for purpose ofclarity, representations and descriptions about those components andprocesses known by those skilled in the art but not necessary tounderstand the present application are omitted in the drawings and thedescription.

As will be appreciated by one skilled in the art, aspects of the presentapplication may be embodied as a system, a device (e.g., a cellulartelephone, a portable media player, a personal computer, a server, atelevision set-top box, or a digital video recorder, or any other mediaplayer), a method or a computer program product. Accordingly, aspects ofthe present application may take the form of an hardware embodiment, ansoftware embodiment (including firmware, resident software, microcodes,etc.) or an embodiment combining both software and hardware aspects thatmay all generally be referred to herein as a “circuit,” “module” or“system.” Furthermore, aspects of the present application may take theform of a computer program product embodied in one or more computerreadable mediums having computer readable program code embodied thereon.

Any combination of one or more computer readable mediums may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electromagnetic or optical signal, or any suitable combination thereof.

A computer readable signal medium may be any computer readable mediumthat is not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wired line, optical fiber cable, RF, etc., or any suitable combinationof the foregoing.

Computer program code for carrying out operations for aspects of thepresent application may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer as a stand-alone software package, or partly on theuser's computer and partly on a remote computer or entirely on theremote computer or server. In the latter scenario, the remote computermay be connected to the user's computer through any type of network,including a local area network (LAN) or a wide area network (WAN), orthe connection may be made to an external computer (for example, throughthe Internet using an Internet Service Provider).

Aspects of the present application are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theapplication. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational operations to be performed on the computer,other programmable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Below will be described in detail the embodiments of the presentapplication. For clarity, the description is organized in the followingarchitecture:

Part 1: Audio Processing Apparatus and Methods

Section 1.1 Audio Types

Section 1.2 Confidence Values of Audio Types and Architecture ofClassifiers

Section 1.3 Smoothing of Confidence Values of Audio Types

Section 1.4 Parameter Adjusting

Section 1.5 Parameter Smoothing

Section 1.6 Transition of Audio Types

Section 1.7 Combination of Embodiments and Application Scenarios

Section 1.8 Audio Processing Method

Part 2: Dialog Enhancer Controller and Controlling Method

Section 2.1 Level of Dialog Enhancement

Section 2.2 Thresholds for Determining Frequency Bands to be Enhanced

Section 2.3 Adjustment to Background Level

Section 2.4 Combination of Embodiments and Application Scenarios

Section 2.5 Dialog Enhancer Controlling Method

Part 3: Surround Virtualizer Controller and Controlling Method

Section 3.1 Surround Boost Amount

Section 3.2 Start Frequency

Section 3.3 Combination of Embodiments and Application Scenarios

Section 3.4 Surround Virtualizer Controlling Method

Part 4: Volume Leveler Controller and Controlling Method

Section 4.1 Informative and Interfering Content Types

Section 4.2 Content Types in Different Contexts

Section 4.3 Context Types

Section 4.4 Combination of Embodiments and Application Scenarios

Section 4.5 Volume Leveler Controlling Method

Part 5: Equalizer Controller and Controlling Method

Section 5.1 Control Based on Content Type

Section 5.2 Likelihood Of Dominant Sources In Music

Section 5.3 Equalizer presets

Section 5.4 Control Based on Context Type

Section 5.5 Combination of Embodiments and Application Scenarios

Section 5.6 Equalizer Controlling Method

Part 6: Audio Classifiers and Classifying Methods

Section 6.1 Context Classifier Based on Content Type Classification

Section 6.2 Extraction of Long-term Features

Section 6.3 Extraction of Short-term Features

Section 6.4 Combination of Embodiments and Application Scenarios

Section 6.5 Audio Classifying Methods

Part 7: VoIP Classifiers and Classifying Methods

Section 7.1 Context Classification Based on Short-term Segment

Section 7.2 Classification Using VoIP Speech and VoIP Noise

Section 7.3 Smoothing Fluctuation

Section 7.4 Combination of Embodiments and Application Scenarios

Section 7.5 VoIP Classifying Methods

Part 1: Audio Processing Apparatus and Methods

FIG. 1 illustrates a general framework of a content-adaptive audioprocessing apparatus 100 that supports automatic configuration of atleast one audio improving device 400 with improved parameters based onthe audio content on playback. It comprises three major components: anaudio classifier 200, an adjusting unit 300 and an audio improvingdevice 400.

The audio classifier 200 is for classifying an audio signal into atleast one audio type in real time. It automatically identifies the audiotypes of the content on playback. Any audio classification technologies,such as through signal processing, machine learning, and patternrecognition, can be applied to identify the audio content. Confidencevalues, which represent the probabilities of the audio content regardinga set of pre-defined target audio types, are estimated generally at thesame time.

The audio improving device 400 is for improving the experience of theaudience by performing processing on the audio signal, and will bediscussed in detail later.

The adjusting unit 300 is for adjusting at least one parameter of theaudio improving device in a continuous manner based on the confidencevalue of the at least one audio type. It is designed to steer thebehavior of the audio improving device 400. It estimates the mostsuitable parameters of the corresponding audio improving device based onthe results obtained from the audio classifier 200.

Various audio improving devices can be applied in this apparatus. FIG. 2shows an example system comprising four audio improving devices,including Dialog Enhancer (DE) 402, Surround Virtualizer (SV) 404,Volume Leveler (VL) 406 and Equalizer (EQ) 408. Each audio improvingdevice can be automatically adjusted in a continuous manner, based onthe results (audio types and/or confidence values) obtained in the audioclassifier 200.

Of course, the audio processing apparatus may not necessarily includeall the kinds of audio improving devices, but may include only one ormore of them. On the other hand, the audio improving devices are notlimited to those devices given in the present disclosure and may includemore kinds of audio improving devices which are also within the scope ofthe present application. Furthermore, the names of those audio improvingdevices discussed in the present disclosure, including Dialog Enhancer(DE) 402, Surround Virtualizer (SV) 404, Volume Leveler (VL) 406 andEqualizer (EQ) 408, shall not constitute a limitation, and each of themshall be construed as covering any other devices realizing the same orsimilar functions.

1.1 Audio Types

For properly controlling various kinds of audio improving device, thepresent application further provides a new architecture of audio types,although those audio types in the prior art are also applicable here.

Specifically, audio types from different semantic levels are modeled,including low-level audio elements representing the fundamentalcomponents in audio signals and high-level audio genres representing themost popular audio contents in real-life user entertainmentapplications. The former may also be nominated as “content type”.Fundamental audio content types may include speech, music (includingsong), background sounds (or sound effects) and noise.

The meaning of speech and music is self-evident. The noise in thepresent application means physical noise, not semantic noise. Physicalnoise in the present application may include the noises from, forexample, air conditioners, and those noises originating from technicalreasons, such as pink noises due to the signal transmitting path. Incontrast, the “background sounds” in the present application are thosesound effects which may be auditory events happening around the coretarget of the listener's attention. For example, in an audio signal in atelephone call, besides the voice of the talker, there may be some othersounds not intended, such as the voices of some other persons irrelevantto the telephone call, sounds of keyboards, sounds of footsteps, and soon. These unwanted sounds are referred to as “background sounds”, notnoise. In other words, we may define “background sounds” as those soundsthat are not the target (or core target of the listener's attention), oreven are not wanted, but still have some semantic meaning; while “noise”may be defined as those unwanted sounds except the target sounds and thebackground sounds.

Sometimes background sounds are really not “unwanted” but createdintentionally and carry some useful information, such as thosebackground sounds in a movie, a TV program or a radio broadcast program.So, sometimes it may also be referred to as “sound effects”. Hereinafterin the present disclosure, only “background sounds” is used forconciseness, and it may be further abbreviated as “background”.

Further, the music may be further classified as music without dominantsources and music with dominant sources. If there is a source (voice oran instrument) is much stronger than the other sources in a music piece,it is referred to as “music with dominant source”; otherwise, it isreferred to as “music without dominant source”. For example, in apolyphonic music accompanied with singing voice and various instruments,if it is harmonically balanced, or the energy of several most salientsources are comparable to each other, it is considered to be a musicwithout dominant source; in contrast, if a source (e.g. voice) is muchlouder while the others are much quieter, it is considered to contain adominant source. As another example, singular or distinctive instrumenttones are “music with dominant source”.

The music may be further classified as different types based ondifferent standards. It can be classified based on genres of the music,such as rock, jazz, rap and folk, but not limited thereto. It can alsobe classified based on instruments, such as vocal music and instrumentalmusic. Instrumental music may include various music played withdifferent instruments, such as piano music and guitar music. Otherexample standards include rhythm, tempo, timbre of the music and/or anyother musical attributes, so that music can be grouped together based onthe similarity of these attributes. For example, according to timbre,vocal music may be classified as tenor, baritone, bass, soprano, mezzosoprano and alto.

The content type of an audio signal may be classified with respect toshort-term audio segments, such as comprised of a plurality of frames.Generally, an audio frame is of a length of multiple milliseconds, suchas 20 ms, and the length of a short-term audio segment to be classifiedby the audio classifier may have a length from several hundred ofmilliseconds to several seconds, such as 1 second.

For controlling the audio improving device in a content-adaptive manner,the audio signal may be classified in real time. For the content typestated above, the content type of the present short-term audio segmentrepresents the content type of the present audio signal. Since thelength of a short-term audio segment is not so long, the audio signalmay be divided as non-overlapped short-term audio segments one afteranother. However, the short-term audio segments may also be sampledcontinuously/semi-continuously along the time line of the audio signal.That is, the short-term audio segments may be sampled with a window witha predetermined length (intended length of the short-term audio segment)moving along the time line of the audio signal at a step size of one ormore frames.

The high-level audio genres may also be nominated as “context type”,since it indicates a long-term type of the audio signal, and may beregarded as an environment or context of the instant sound event, whichmay be classified into the content types as stated above. According tothe present application, the context type may include the most popularaudio applications, such as movie-like media, music (including song),game and VoIP (Voice on Internet Protocol).

The meaning of music, game and VoIP is self-evident. Movie-like mediamay include movie, TV program, radio broadcast program or any otheraudio media similar to aforementioned. The main characteristic ofmovie-like media is a mixture of possible speeches, music and variouskinds of background sounds (sound effects).

It may be noted both the content type and the context type include music(including song). Hereinafter in the present application, we use thewordings “short-term music” and “long-term music” to distinguish themrespectively.

For some embodiments of the present application, some other context typearchitectures are also proposed.

For example, an audio signal may be classified as high-quality audio(such as movie-like media and music CD) or low-quality audio (such asVoIP, low bit rate online streaming audio and user generated content),which may be collectively referred to as “audio quality types”.

As another example, an audio signal may be classified as VoIP ornon-VoIP, which may be regarded as a transform of the 4-context typearchitecture mentioned above (VoIP, movie-like media, (long-term) musicand game). In connection with the context of VoIP or non-VoIP, an audiosignal may be classified as VoIP-related audio content types, such asVoIP speech, non-VoIP speech, VoIP noise and non-VoIP noise. Thearchitecture of VoIP audio content types are especially useful todifferentiate VoIP and non-VoIP contexts since VoIP context is usuallythe most challenging application scenario of a volume leveler (one kindof audio improving device).

Generally, the context type of an audio signal may be classified withrespect to long-term audio segments longer than the short-term audiosegments. A long-term audio segment is comprised of a plurality offrames in a number more than the number of frames in a short-term audiosegment. A long-term audio segment may also be comprised of a pluralityof short-term audio segments. Generally, a long-term audio segment mayhave a length in the order of seconds, such as several seconds toseveral tens of seconds, say 10 seconds.

Similarly, for controlling the audio improving device in an adaptivemanner, the audio signal may be classified into context types in realtime. Similarly, the context type of the present long-term audio segmentrepresents the context type of the present audio signal. Since thelength of a long-term audio segment is relatively long, the audio signalmay be sampled continuously/semi-continuously along the time line of theaudio signal to avoid abrupt change of its context type and thus abruptchange of the working parameters of the audio improving device(s). Thatis, the long-term audio segments may be sampled with a window with apredetermined length (intended length of the long-term audio segment)moving along the time line of the audio signal at a step size of one ormore frames, or one or more short-term segments.

Above have been described both the content type and the context type. Inthe embodiments of the present application, the adjusting unit 300 mayadjust at least one parameter of the audio improving device(s) based onat least one of the various content types, and/or at least one of thevarious context types. Therefore, as shown in FIG. 3, in a variant ofthe embodiment shown in FIG. 1, the audio classifier 200 may compriseeither an audio content classifier 202 or an audio context classifier204, or both.

Above have been mentioned different audio types based on differentstandards (such as for the context types), as well as different audiotypes on different hierarchical levels (such as for the content types).However, the standards and the hierarchical levels are just for theconvenience of description here and are definitely not limiting. Inother words, in the present application, any two or more of the audiotypes mentioned above can be identified by the audio classifier 200 atthe same time and be considered by the adjusting unit 300 at the sametime, as will be described later. In other words, all the audio types indifferent hierarchical levels may be parallel, or in the same level.

1.2 Confidence Values of Audio Types and Architecture of Classifiers

The audio classifier 200 may output hard-decision results, or theadjusting unit 300 may regard the results of the audio classifier 200 ashard-decision results. Even for hard decision, multiple audio types canbe assigned to an audio segment. For example, an audio segment can belabeled by both ‘speech’ and ‘short-term music’ since it may be amixture signal of speech and short-term music. The obtained labels canbe directly used to steer the audio improving device(s) 400. A simpleexample is to enable the dialog enhancer 402 when speech is present andturn it off when speech is absent. However, this hard decision methodmay introduce some unnaturalness at the transition points from one audiotype to another, if without careful smoothing scheme (which will bediscussed later).

In order to have more flexibility and tune the parameters of the audioimproving devices in a continuous manner, the confidence value of eachtarget audio type can be estimated (soft decision). A confidence valuerepresents the matched level between the to-be-identified audio contentand the target audio type, with values from 0 to 1.

As stated before, many classifying techniques may output confidencevalues directly. The confidence value can also be calculated fromvarious methods, which may be regarded as a part of the classifier. Forexample, if the audio models are trained by some probabilistic modelingtechnologies such as Gaussian Mixture Models (GMM), posteriorprobability can be used to represent confidence value, as

$\begin{matrix}{{p( c_{i} \middle| x )} = \frac{p( x \middle| c_{i} )}{\sum\limits_{i = 1}^{N}\;{p( x \middle| c_{i} )}}} & (1)\end{matrix}$where x is a piece of audio segment, c_(i) is a target audio type, N isthe number of target audio types, p(x|c_(i)) is the likelihood that theaudio segment x is of the audio type c_(i), and p(c_(i)|x) is thecorresponding posterior probability.

On the other hand, if the audio models are trained from somediscriminative methods like Support Vector Machine (SVM) and adaBoost,only scores (real values) are obtained from model comparison. In thesecases, a sigmoid function is usually used to map the obtained score(theoretically from −∞ to ∞) to the expected confidence (from 0 to 1):

$\begin{matrix}{{conf} = \frac{1}{1 + e^{{Ay} + B}}} & (2)\end{matrix}$where the y is the output score from SVM or adaBoost, A and B are twoparameters need to be estimated from a training data set by using somewell-known technologies.

For some embodiments of the present application, the adjusting unit 300may use more than two content types and/or more than two context types.Then, the audio content classifier 202 need identify more than twocontent types and/or the audio context classifier 204 need identify morethan two context types. In such a situation, either the audio contentclassifier 202 or the audio context classifier 204 may be a group ofclassifiers organized in certain architecture.

For example, if the adjusting unit 300 needs all the four kinds ofcontext types movie-like media, long-term music, game and VoIP, then theaudio context classifier 204 may have the following differentarchitectures:

First, the audio context classifier 204 may comprise 6 one-to-one binaryclassifiers (each classifier discriminates one target audio type fromanother target audio type) organized as shown in FIG. 4, 3 one-to-othersbinary classifiers (each classifier discriminates a target audio typefrom the others) organized as shown in FIG. 5, and 4 one-to-othersclassifiers organized as shown in FIG. 6. There are also otherarchitectures such as Decision Directed Acyclic Graph (DDAG)architecture. Note that in FIGS. 4-6 and the corresponding descriptionbelow, “movie” instead of “movie-like media” is used for conciseness.

Each binary classifier will give a confidence score H(x) for its output(x represents an audio segment). After the outputs of each binaryclassifier are obtained, we need map them to the final confidence valuesof the identified context types.

Generally, assuming the audio signal is to be classified into M contexttypes (M is a positive integer). The conventional one-to-onearchitecture constructs

(

−1)/2 classifiers where each one is trained on data from two classes,then each one-to-one classifier casts one vote for its preferred class,and the final result is the class with the most votes among the

(

−1)/2 classifiers' classifications. Comparing to the conventionalone-to-one architecture, the hierarchical architecture in FIG. 4 alsoneeds to construct

(

−1)/2 classifiers. However, the testing iterations can be shortened to

−1, as the segment x will be determined to be/not be in thecorresponding class at each hierarchy level and the overall level countis

−1. The final confidence values for various context types may becalculated from the binary classification confidence H_(k)(x), forexample (k=1, 2, . . . 6, representing different context types):C _(MOVIE)=(1−H ₁(x))·(1−H ₃(x))·(1−H ₆(x))C _(VOIP) =H ₁(x)·H ₂(x)·H ₄(x)C _(MUSIC) =H ₁(x)·(1−H ₂(x))·(1−H ₅(x))+H ₃(x)·(1−H ₁(x))·(1−H ₅(x))+H₆(x)·(1−H ₁(x))·(1−H ₃(x))C _(GAME) =H ₁(x)·H ₂(x)·(1−H ₄(x))+H ₁(x)·H ₅(x)·(1−H ₂(x))+H ₃(x)·H₅(x)·(1−H ₁(x))

In the architecture shown in FIG. 5, the mapping function from thebinary classification results H_(k)(x) to the final confidence valuescan be defined as the following example:C _(MOVIE) =H ₁(x)C _(MUSIC) =H ₂(x)·(1−H ₁(x))C _(VOIP) =H ₃(x)·(1−H ₂(x))·(1−H ₁(x))C _(GAME)=(1−H ₃(x))·(1−H ₂(x))·(1−H ₁(x))

In the architecture illustrated in FIG. 6, the final confidence valuescan be equal to the corresponding binary classification resultsH_(k)(x), or if the sum of the confidence values for all classes isrequired to be 1, then the final confidence values can be simplynormalized based on the estimated H_(k)(x):C _(MOVIE) =H ₁(x)/(H ₁(x)+H ₂(x)+H ₃(x)+H ₄(x))C _(MUSIC) =H ₂(x)/(H ₁(x)+H ₂(x)+H ₃(x)+H ₄(x))C _(VOIP) =H ₃(x)/(H ₁(x)+H ₂(x)+H ₃(x)+H ₄(x))C _(GAME) =H ₄(x)/(H ₁(x)+H ₂(x)+H ₃(x)+H ₄(x))

The one or more with the maximum confidence values can be determined tobe the final identified class.

It should be noted that in the architectures shown in FIGS. 4-6, thesequence of different binary classifiers are not necessarily as shown,but may be other sequences, which may be selected by manual assigning orautomatic learning according to different requirements of variousapplications.

The descriptions above are directed to audio context classifiers 204.For the audio content classifier 202, the situation is similar.

Alternatively, either the audio content classifier 202 or the audiocontext classifier 204 may be implemented as one single classifieridentifying all the content types/context types at the same time, andgive the corresponding confidence values at the same time. There aremany existing techniques for doing this.

Using the confidence value, the output of the audio classifier 200 canbe represented as a vector, with each dimension representing theconfidence value of each target audio type. For example, if the targetaudio types are (speech, short-term music, noise, background)sequentially, an example output result could be (0.9, 0.5, 0.0, 0.0),indicating that it is 90% sure the audio content is speech, and 50% surethe audio is music. It is noted that the sum of all the dimensions inthe output vector is not necessary to be one (for example, the resultsfrom FIG. 6 are not necessary normalized), meaning that the audio signalmay be a mixture signal of speech and short-term music.

Later in Part 6 and Part 7, a novel implementation of the audio contextclassification and the audio content classification will be discussed indetail.

1.3 Smoothing of Confidence Values of Audio Types

Optionally, after each audio segment has been classified into thepredefined audio types, an additional step is to smooth theclassification results along time line to avoid abrupt jump from onetype to another and to make more smooth estimation of the parameters inthe audio improving devices. For example, a long excerpt is classifiedas movie-like media except for only one segment classified as VoIP, thenthe abrupt VoIP decision can be revised to movie-like media bysmoothing.

Therefore, in a variant of the embodiment as shown in FIG. 7, a typesmoothing unit 712 is further provided for, for each audio type,smoothing the confidence value of the audio signal at the present time.

A common smoothing method is based on weighted average, such ascalculates a weighted sum of the actual confidence value at the presentand a smoothed confidence value of the last time, as follows:smoothConf(t)=β·smoothConf(t−1)+(1−β)·conf(t)  (3)where t represents the present time (the present audio segment), t−1represents the last time (the last audio segment), β is the weight, confand smoothConf are the confidence values before and after smoothing,respectively.

From the confidence values' point of view, the results from harddecision of the classifiers can be also represented with confidencevalues, with the values being either 0 or 1. That is, if a target audiotype is chosen and assigned to an audio segment, the correspondingconfidence is 1; otherwise, the confidence is 0. Therefore, even if theaudio classifier 200 does not give the confidence value but just gives ahard decision regarding the audio type, continuous adjustment of theadjusting unit 300 is still possible through the smoothing operation ofthe type smoothing unit 712.

The smoothing algorithm can be ‘asymmetric’, by using differentsmoothing weight for different cases. For example, the weights forcalculating the weighted sum may be adaptively changed based onconfidence value of the audio type of the audio signal. The confidencevalue of the present segment is larger, is larger its weight.

From another point of view, the weights for calculating the weighted summay be adaptively changed based on different transition pairs from oneaudio type to another audio type, especially when the audio improvingdevice(s) is adjusted based on multiple content types as identified bythe audio classifier 200, instead of based on the presence or absence ofone single content type. For example, for a transition from an audiotype more frequently appearing in certain context to another audio typenot so frequently appearing in the context, the confidence value of thelatter may be smoothed so that it will not increase so fast, because itmight just be an occasional interruption.

Another factor is the changing (increasing or decreasing) trend,including the changing rate. Suppose we care more about the latency whenan audio type becomes present (that is, when its confidence valueincreases), we can design the smoothing algorithm in the following way:

$\begin{matrix}{{{smoothConf}(t)} = \{ \begin{matrix}{{conf}(t)} & {{{conf}(t)} \geq {{smoothConf}( {t - 1} )}} \\{{\beta \cdot {{smoothConf}( {t - 1} )}} + {( {1 - \beta} ) \cdot {{conf}(t)}}} & {otherwise}\end{matrix} } & (4)\end{matrix}$

The above formula allows the smoothed confidence value quickly respondsto the current state when confidence value increases and slowly smoothaway when confidence value decreases. Variants of the smoothingfunctions can be easily designed in the similar way. For example, theformula (4) may be revised so that the weight of conf(t) becomes largerwhen conf(t)>=smoothConf(t−1). In fact, in formula (4), it can beregarded that β=0 and the weight of conf(t) becomes the largest, that is1.

From a different point of view, considering the changing trend ofcertain audio type is just a specific example of considering differenttransition pairs of audio types. For example, increasing of theconfidence value of type A may be regarded as transition from non-A toA, and decreasing of the confidence value of type A may be regarded astransition of A to non-A.

1.4 Parameter Adjusting

The adjusting unit 300 is designed to estimate or adjust properparameters for the audio improving device(s) 400 based on the obtainedresults from the audio classifier 200. Different adjusting algorithmsmay be designed for different audio improving devices, by using eitherthe content type or context type, or both for a joint decision. Forexample, with context type information such as movie-like media andlong-term music, the presets as mentioned before can be automaticallyselected and applied on the corresponding content. With the content typeinformation available, the parameters of each audio improving device canbe tuned in a finer manner, as shown in the subsequent parts. Thecontent type information and the context information can be furtherjointly used in the adjusting unit 300 to balance the long-term andshort-term information. The specific adjusting algorithm for a specificaudio improving device may be regarded as a separate adjusting unit, orthe different adjusting algorithms may be collectively regarded as aunited adjusting unit.

That is, the adjusting unit 300 may be configured to adjust the at leastone parameter of the audio improving device based on the confidencevalue of at least one content type and/or the confidence value of atleast one context type. For a specific audio improving device, some ofthe audio types are informative, and some of the audio types areinterfering. Accordingly, the parameters of the specific audio improvingdevice may be either positively or negatively correlate to theconfidence value(s) of the informative audio type(s) or the interferingaudio type(s). Here, “positively correlate” means the parameterincreases or decreases with the increasing or decreasing of theconfidence value of the audio type, in a linear manner or in anon-linear manner. “Negatively correlate” means the parameter increasesor decreases with, respectively, the decreasing or increasing of theconfidence value of the audio type, in a linear manner or in anon-linear manner.

Here, the decreasing and increasing of the confidence value are directly“transferred” to the parameters to be adjusted by the positive ornegative correlation. In mathematics, such correlation or “transfer” maybe embodied as linear proportion or inverse proportion, plus or minus(addition or subtraction) operation, multiplying or dividing operationor non-linear function. All these forms of correlation may be referredto as “transfer function”. To determine increasing or decreasing of theconfidence value, we can also compare the present confidence value orits mathematical transform with the last confidence value or a pluralityof history confidence values, or their mathematical transforms. In thecontext of the present application, the term “compare” means eithercomparison through subtraction operation or comparison through divisionoperation. We can determine an increase or decrease by determiningwhether the difference is greater than 0 or whether the ratio is greaterthan 1.

In specific implementations, we can directly relate the parameters withthe confidence values or their ratios or differences through properalgorithm (such as transfer function) and it is not necessary for an“external observer” to know explicitly whether a specific confidencevalue and/or a specific parameter has increased or decreased. Somespecific examples will be given in subsequent Parts 2-5 about specificaudio improving devices.

As stated in the previous section, with respect to the same audiosegment, the classifier 200 may identify multiple audio types withrespective confidence values, which confidence values may notnecessarily amount to 1, since the audio segment may comprise multiplecomponents at the same time, such as music and speech and backgroundsounds. In such a situation, the parameters of the audio improvingdevices shall be balanced between different audio types. For example,the adjusting unit 300 may be configured to consider at least some ofthe multiple audio types through weighting the confidence values of theat least one audio type based on the importance of the at least oneaudio type. More important is a specific audio type, more is theparameters influenced thereby.

The weight can also reflect informative and interfering effect of theaudio type. For example, for an interfering audio type, a minus weightmay be given. Some specific examples will be given in the subsequentParts 2-5 about specific audio improving devices.

Please note in the context of the present application, “weight” has abroader meaning than coefficients in a multinomial. Besides thecoefficients in a multinomial, it can also take the form of exponent orpower. When being the coefficients in a multinomial, the weightingcoefficients may be or may be not normalized. In brief, the weight justrepresents how much influence the weighted object has upon the parameterto be adjusted.

In some other embodiments, for the multiple audio types contained in thesame audio segment, the confidence values thereof may be converted toweights through being normalized, then the final parameter may bedetermined through calculating a sum of parameter preset valuespredefined for each audio type and weighted by the weights based on theconfidence values. That is, the adjusting unit 300 may be configured toconsider the multiple audio types through weighting the effects of themultiple audio types based on the confidence values.

As a specific example of weighting, the adjusting unit is configured toconsider at least one dominant audio type based on the confidencevalues. For those audio types having too low confidence values (lessthan a threshold), they may not be considered. This is equivalent tothat the weights of the other audio types the confidence values of whichare less than the threshold are set as zero. Some specific examples willbe given in the subsequent Parts 2-5 about specific audio improvingdevices.

The content type and the context type can be considered together. In oneembodiment, they can be regarded as on the same level and theirconfidence values may have respective weights. In another embodiment,just as the nomination shows, the “context type” is the context orenvironment where the “content type” is located, and thus the adjustingunit 200 may be configured so that the content type in an audio signalof a different context type is assigned a different weight depending onthe context type of the audio signal. Generally speaking, any audio typecan constitute a context of another audio type, and thus the adjustingunit 200 may be configured to modify the weight of one audio type withthe confidence value of another audio type. Some specific examples willbe given in the subsequent Parts 2-5 about specific audio improvingdevices.

In the context of the present application, “parameter” has a broadermeaning than its literal meaning. Besides a parameter having one singlevalue, it may also means a preset as mentioned before, including a setof different parameters, a vector comprised of different parameters, ora profile. Specifically, in the subsequent Parts 2-5 the followingparameters will be discussed but the present application is not limitedthereto: the level of dialog enhancement, the thresholds for determiningfrequency bands to be dialog-enhanced, the background level, thesurround boost amount, the start frequency for the surround virtualizer,the dynamic gain or the range of the dynamic gain of a volume leveler,the parameters indicating the degree of the audio signal being a newperceptible audio event, the equalization level, equalization profilesand spectral balance presets.

1.5 Parameter Smoothing

In Section 1.3, we have discussed smoothing the confidence value of anaudio type to avoid its abrupt change, and thus avoid abrupt change ofthe parameters of the audio improving device(s). Other measures are alsopossible. One is to smooth the parameter adjusted based on the audiotype, and will be discussed in this section; the other is to configurethe audio classifier and/or the adjusting unit to delay the change ofthe results of the audio classifier, and this will be discussed inSection 1.6.

In one embodiment, the parameter can be further smoothed to avoid rapidchange which may introduce audible artifacts at transition points, as{tilde over (L)}(t)=τ{tilde over (L)}(t−1)+(1−τ)L(t)  (3′)

where {tilde over (L)}(t) is the smoothed parameter, L(t) is thenon-smoothed parameter τ is a coefficient representing a time constant,t is the present time and t−1 is the last time.

That is, as shown in FIG. 8 the audio processing apparatus may comprisea parameter smoothing unit 814 for, for a parameter of the audioimproving device (such as at least one of the dialog enhancer 402, thesurround virtualizer 404, the volume leveler 406 and the equalizer 408)adjusted by the adjusting unit 300, smoothing the parameter valuedetermined by the adjusting unit 300 at the present time by calculatinga weighted sum of the parameter value determined by the adjusting unitat the present time and a smoothed parameter value of the last time.

The time constant τ can be a fixed value based on the specificrequirement of an application and/or the implementation of the audioimproving device 400. It could be also adaptively changed based on theaudio type, especially based on the different transition types from oneaudio type to another, such as from music to speech, and from speech tomusic.

Take equalizer as an example (further details may be referred to in Part5). Equalization is good to apply on music content but not on speechcontent. Thus, for smoothing the level of equalization, the timeconstant can be relatively small when the audio signal transits frommusic to speech, so that a smaller equalization level can be applied onspeech content more quickly. On the other hand, the time constant forthe transition from speech to music can be relatively large in order toavoid the audible artifacts at the transition points.

To estimate the transition type (e.g., from speech to music or frommusic to speech), the content classification results can be useddirectly. That is, classifying the audio content into either music orspeech makes it straightforward to get the transition type. To estimatethe transition in a more continuous manner, we can also rely on theestimated unsmoothed equalization level, instead of directly comparingthe hard decisions of the audio types. The general idea is, if theunsmoothed equalization level is increasing, it indicates a transitionfrom speech to music (or more music like); otherwise, it is more like atransition from music to speech (or more speech like). Bydifferentiating different transition types, the time constant can be setcorrespondingly, one example is:

$\begin{matrix}{{\tau(t)} = \{ \begin{matrix}\tau_{1} & {{L(t)} \geq {L( {t - 1} )}} \\\tau_{2} & {{L(t)} < {L( {t - 1} )}}\end{matrix} } & ( 4^{\prime} )\end{matrix}$where τ (t) is the time-variant time constant depending on the content,τ1 and τ2 are two preset time constant values, usually satisfying τ1>τ2.Intuitively, the above function indicates a relatively slow transitionwhen the equalization level is increasing, and a relatively fasttransition when equalization level is decreasing, but the presentapplication is not limited thereto. Further, the parameter is notlimited to the equalization level, but may be other parameters. That is,the parameter smoothing unit 814 may be configured so that the weightsfor calculating the weighted sum are adaptively changed based on anincreasing or decreasing trend of the parameter value determined by theadjusting unit 300.1.6 Transition of Audio Types

With reference to FIGS. 9 and 10 will be described another scheme foravoiding abrupt change of audio type, and thus avoiding abrupt change ofthe parameters of the audio improving device(s).

As shown in FIG. 9, the audio processing apparatus 100 may furthercomprise a timer 916 for measuring the lasting time during which theaudio classifier 200 continuously outputs the same new audio type,wherein the adjusting unit 300 may be configured to continue to use thepresent audio type until the length of the lasting time of the new audiotype reaches a threshold.

In other words, an observation (or sustaining) phase is introduced, asillustrated in FIG. 10. With the observation phase (corresponding to thethreshold of the length of the lasting time), the change of audio typeis further monitored for a consecutive amount of time to confirm if theaudio type has really changed, before the adjusting unit 300 really usesthe new audio type.

As shown in FIG. 10, the arrow (1) illustrates the situation where thecurrent state is type A and the result of the audio classifier 200 doesnot change.

If the current state is type A and the result of the audio classifier200 becomes type B, then the timer 916 starts timing, or, as shown inFIG. 10, the process enters an observation phase (the arrow (2)), and aninitial value of the hangover count cnt is set, indicating the amount ofobservation duration (equal to the threshold).

Then, if the audio classifier 200 continuously output type B, then cntcontinuously decreases (the arrow (3)) until cnt is equal to 0 (that is,the length of the lasting time of the new type B reaches the threshold),then the adjusting unit 300 may use the new audio type B (the arrow(4)), or in other words, only up to now may the audio type be regardedhaving really changed to the type B.

Otherwise, if before the cnt becomes zero (before the length of thelasting time reaches the threshold) the output of the audio classifier200 becomes back to the old type A, then the observation phase isterminated and the adjusting unit 300 still uses the old type A (thearrow (5)).

The change from type B to type A may be similar to the process describedabove.

In the above process, the threshold (or the hangover count) may be setbased on the application requirement. It can be a predefined fixedvalue. It can be also adaptively set. In one variant, the threshold isdifferent for different transition pairs from one audio type to anotheraudio type. For example, when changing from type A to type B, thethreshold may be a first value; and when changing from type B to type A,the threshold may be a second value.

In another variant, the hangover count (threshold) may be negativelycorrelated with the confidence value of the new audio type. The generalidea is, if the confidence shows confusing between two types (e.g., whenthe confidence value is only around 0.5), the observation duration needsto be long; otherwise, the duration can be relatively short. Followingthis guideline, an example hangover count can be set by the followingformula,HangCnt=C·|0.5−Conf|+Dwhere HangCnt is the hangover duration or the threshold, C and D are twoparameters that can be set based on the application requirement, usuallyC is negative while D is a positive value.

Incidentally, the timer 916 (and thus the transition process describedabove) has been described above as a part of the audio processingapparatus but outside of the audio classifier 200. In some otherembodiments, it may be regarded as a part of the audio classifier 200,just as described in Section 7.3.

1.7 Combination of Embodiments and Application Scenarios

All the embodiments and variants there of discussed above may beimplemented in any combination thereof, and any components mentioned indifferent parts/embodiments but having the same or similar functions maybe implemented as the same or separate components.

Specifically, when describing the embodiments and their variationshereinbefore, those components having reference signs similar to thosealready described in previous embodiments or variants are omitted, andjust different components are described. In fact, these differentcomponents can either be combined with the components of otherembodiments or variants, or constitute separate solutions alone. Forexample, any two or more of the solutions described with reference toFIGS. 1 to 10 may be combined with each other. As the most completesolution, the audio processing apparatus may comprise both the audiocontent classifier 202 and the audio context classifier 204, as well asthe type smoothing unit 712, the parameter smoothing unit 814 and thetimer 916.

As mentioned before, the audio improving devices 400 may include thedialog enhancer 402, the surround virtualizer 404, the volume leveler406 and the equalizer 408. The audio processing apparatus 100 mayinclude any one or more of them, with the adjusting unit 300 adapted tothem. When involving multiple audio improving devices 400, the adjustingunit 300 may be regarded as including multiple sub-units 300A to 300D(FIGS. 15, 18, 20 and 22) specific to respective audio improving devices400, or still be regarded as one united adjusting unit. When specific toan audio improving device, the adjusting unit 300 together with theaudio classifier 200, as well as other possible components, may beregarded as the controller of the specific audio improving device, whichwill be discussed in detail in subsequent Parts 2-5.

In addition, the audio improving devices 400 are not limited to theexamples as mentioned and may include any other audio improving device.

Further, any solutions already discussed or any combinations thereof maybe further combined with any embodiment described or implied in theother parts of this disclosure. Especially, the embodiments of the audioclassifiers as will be discussed in Parts 6 and 7 may be used in theaudio processing apparatus.

1.8 Audio Processing Method

In the process of describing the audio processing apparatus in theembodiments hereinbefore, apparently disclosed are also some processesor methods. Hereinafter a summary of these methods is given withoutrepeating some of the details already discussed hereinbefore, but itshall be noted that although the methods are disclosed in the process ofdescribing the audio processing apparatus, the methods do notnecessarily adopt those components as described or are not necessarilyexecuted by those components. For example, the embodiments of the audioprocessing apparatus may be realized partially or completely withhardware and/or firmware, while it is possible that the audio processingmethod discussed below may be realized totally by a computer-executableprogram, although the methods may also adopt the hardware and/orfirmware of the audio processing apparatus.

The methods will be described below with reference to FIGS. 11-14.Please note that in correspondence to the streaming property of theaudio signal, the various operations are repeated when the method isimplemented in real time, and different operations are not necessarilywith respect to the same audio segment.

In an embodiment as shown in FIG. 11, an audio processing method isprovided. First, the audio signal to be processed is classified into atleast one audio type in real time (operation 1102). Based on theconfidence value of the at least one audio type, at least one parameterfor audio improvement can be continuously adjusted (operation 1104). Theaudio improvement may be dialog enhancing (operation 1106), surroundvirtualizing (operation 1108), volume leveling (1110) and/or equalizing(operation 1112). Correspondingly, the at least one parameter maycomprise at least one parameter for at least one of dialog enhancementprocessing, surround virtualizing processing, volume leveling processingand equalizing processing.

Here, “in real time” and “continuously” means the audio type, and thusthe parameter will change in real time with the specific content of theaudio signal, and “continuously” also means the adjustment is acontinuous adjustment based on the confidence value, rather than abruptor discrete adjustment.

The audio type may comprise content type and/or context type.Correspondingly, the operation 1104 of adjusting may be configured toadjust the at least one parameter based on the confidence value of atleast one content type and the confidence value of at least one contexttype. The content type may further comprise at least one of contenttypes of short-term music, speech, background sound and noise. Thecontext type may further comprise at least one of context types oflong-term music, movie-like media, game and VoIP.

Some other context type schemes are also proposed, like VoIP relatedcontext types including VoIP and non-VoIP, and audio quality typesincluding high-quality audio or low-quality audio.

The short-term music may be further classified into sub-types accordingto different standards. Depending on the presence of dominant source, itmay comprise music without dominant sources and music with dominantsources. In addition, the short-term music may comprise at least onegenre-based cluster or at least one instrument-based cluster or at leastone music cluster classified based on rhythm, tempo, timbre of musicand/or any other musical attributes.

When both content types and the context types are identified, theimportance of a content type may be determined by the context type wherethe content type is located. That is, the content type in an audiosignal of a different context type is assigned a different weightdepending on the context type of the audio signal. More generally, oneaudio type may influence or may be premise of another audio type.Therefore, the operation of adjusting 1104 may be configured to modifythe weight of one audio type with the confidence value of another audiotype.

When an audio signal is classified into multiple audio types at the sametime (that is with respect to the same audio segment), the operation ofadjusting 1104 may consider some or all of the identified audio typesfor adjusting the parameter(s) for improving that audio segment. Forexample, the operation of adjusting 1104 may be configured to weight theconfidence values of the at least one audio type based on the importanceof the at least one audio type. Or, the operation of adjusting 1104 maybe configured to consider at least some of the audio types throughweighting them based on their confidence values. In a special case, theoperation of adjusting 1104 may be configured to consider at least onedominant audio type based on the confidence values.

For avoiding abrupt changes of results, smoothing schemes may beintroduced.

The adjusted parameter value may be smoothed (operation 1214 in FIG.12). For example, the parameter value determined by the operation ofadjusting 1104 at the present time may be replaced with a weighted sumof the parameter value determined by the operation of adjusting at thepresent time and a smoothed parameter value of the last time. Thus,through the iterated smoothing operation, the parameter value issmoothed on the time line.

The weights for calculating the weighted sum may be adaptively changedbased on the audio type of the audio signal, or based on differenttransition pairs from one audio type to another audio type.Alternatively, the weights for calculating the weighted sum areadaptively changed based on an increasing or decreasing trend of theparameter value determined by the operation of adjusting.

Another smoothing scheme is shown in FIG. 13. That is, the method mayfurther comprise, for each audio type, smoothing the confidence value ofthe audio signal at the present time by calculating a weighted sum ofthe actual confidence value at the present and a smoothed confidencevalue of the last time (operation 1303). Similarly to the parametersmoothing operation 1214, the weights for calculating the weighted summay be adaptively changed based on confidence value of the audio type ofthe audio signal, or based on different transition pairs from one audiotype to another audio type.

Another smoothing scheme is a buffer mechanism for delaying thetransition from one audio type to another audio type even if the outputof the audio classifying operation 1102 changes. That is, the operationof adjusting 1104 do not use the new audio type immediately but wait forthe stabilization of the output of the audio classifying operation 1102.

Specifically, the method may comprise measuring the lasting time duringwhich the classifying operation continuously outputs the same new audiotype (operation 1403 in FIG. 14), wherein the operation of adjusting1104 is configured to continue to use the present audio type (“N” inoperation 14035 and operation 11041) until the length of the lastingtime of the new audio type reaches a threshold (“Y” in operation 14035and operation 11042). Specifically, when the audio type output from theaudio classifying operation 1102 changes with respect to the presentaudio type used in the audio parameter adjusting operation 1104 (“Y” inoperation 14031), then the timing starts (operation 14032). If the audioclassifying operation 1102 continues to output the new audio type, thatis, if the judgment in operation 14031 continues to be “Y”, then thetiming continues (operation 14032). Finally when the lasting time of thenew audio type reaches a threshold (“Y” in operation 14035), theadjusting operation 1104 uses the new audio type (operation 11042), andthe timing is reset (operation 14034) for preparing for the next switchof the audio type. Before reaching the threshold (“N” in operation14035), the adjusting operation 1104 continues to use the present audiotype (operation 11041).

Here the timing may be implemented with the mechanism of a timer(counting up or counting down). If after the timing starts but beforereaching the threshold, the output of the audio classifying operation1102 becomes back to the present audio type used in the adjustingoperation 1104, it should be regarded that there is no change (“N” inoperation 14031) with respect to the present audio type used by theadjusting operation 1104. But the present classification result(corresponding to the present audio segment to be classified in theaudio signal) changes with respect to the previous output (correspondingto the previous audio segment to be classified in the audio signal) ofthe audio classifying operation 1102 (“Y” in operation 14033), thus thetiming is reset (operation 14034), until the next change (“Y” inoperation 14031) starts the timing. Of course, if the classificationresult of the audio classifying operation 1102 does not change withrespect to the present audio type used by the audio parameter adjustingoperation 1104 (“N” in operation 14031), nor changes with respect to theprevious classification (“N” in operation 14033), it shows the audioclassification is in a stable state and the present audio type continuesto be used.

The threshold used here may also be different for different transitionpairs from one audio type to another audio type, because when the stateis not so stable, generally we may prefer the audio improving device isin its default conditions rather than others. On the other hand, if theconfidence value of the new audio type is relatively high, it is saferto transit to the new audio type. Therefore, the threshold may benegatively correlated with the confidence value of the new audio type.Higher is the confidence value, lower is the threshold, meaning theaudio type may transit to the new one faster.

Similar to the embodiments of the audio processing apparatus, anycombination of the embodiments of the audio processing method and theirvariations are practical on one hand; and on the other hand, everyaspect of the embodiments of the audio processing method and theirvariations may be separate solutions. Especially, in all the audioprocessing methods, the audio classifying methods as discussed in Parts6 and 7 may be used.

Part 2: Dialog Enhancer Controller and Controlling Method

One example of the audio improving device is Dialog Enhancer (DE), whichaims at continually monitoring the audio on playback, detecting thepresence of dialog, and enhancing the dialog to increase their clarityand intelligibility (making the dialog easier to be heard andunderstood), especially for elders with decreasing hearing capability.Besides detecting if a dialog is present, the frequencies most importantto intelligibility are also detected if a dialog is present and thencorrespondingly enhanced (with dynamic spectral rebalancing). An exampledialog enhancement method is presented in H. Muesch. “Speech Enhancementin Entertainment Audio” published as WO 2008/106036 A2, the entirety ofwhich is incorporated herein by reference.

A common manual configuration on Dialog Enhancer is that it is usuallyenabled on movie-like media content but disabled on music content,because dialog enhancement may falsely trigger too much on musicsignals.

With audio type information available, the level of dialog enhancementand other parameters can be tuned based on the confidence values of theidentified audio types. As a specific example of the audio processingapparatus and method discussed earlier, the dialog enhancer may make useof all the embodiments discussed in Part 1 and any combinations of thoseembodiments. Specifically, in the case of controlling the dialogenhancer, the audio classifier 200 and the adjusting unit 300 in theaudio processing apparatus 100 as shown in FIGS. 1-10 may constitute adialog enhancer controller 1500 as shown in FIG. 15. In this embodiment,since the adjusting unit is specific to the dialog enhancer, it may bereferred to as 300A. And, as discussed in the previous part, the audioclassifier 200 may comprise at least one of the audio content classifier202 and the audio context classifier 204, and the dialog enhancercontroller 1500 may further comprise at least one of the type smoothingunit 712, the parameter smoothing unit 814 and the timer 916.

Therefore, in this part, we will not repeat those contents alreadydescribed in the previous part, and just give some specific examplesthereof.

For a dialog enhancer, the adjustable parameters include but are notlimited to the level of dialog enhancement, the background level, andthe thresholds for determining frequency bands to be enhanced. See H.Muesch. “Speech Enhancement in Entertainment Audio” published as WO2008/106036 A2, the entirety of which is incorporated herein byreference.

2.1 Level of Dialog Enhancement

When involving the level of dialog enhancement, the adjusting unit 300Amay be configured to positively correlate the level of dialogenhancement of the dialog enhancer with the confidence value of speech.Additionally or alternatively, the level may be negatively correlated tothe confidence value of the other content types. Thus, the level ofdialog enhancement can be set to be proportional (linearly ornon-linearly) to the speech confidence, so that the dialog enhancementis less effective in non-speech signals, such as music and backgroundsound (sound effects).

As to the context type, the adjusting unit 300A may be configured topositively correlate the level of dialog enhancement of the dialogenhancer with the confidence value of movie-like media and/or VoIP, andor negatively correlate the level of dialog enhancement of the dialogenhancer with the confidence value of the long-term music and/or game.For example, the level of dialog enhancement can be set to beproportional (linearly or non-linearly) the confidence value ofmovie-like media. When the movie-like media confidence value is 0 (e.g.in the music content), the level of dialog enhancement is also 0, whichis equivalent to disable the dialog enhancer.

As described in the previous part, the content type and the context typemay be considered jointly.

2.2 Thresholds for Determining Frequency Bands to be Enhanced

During the working of the dialog enhancer, there is a threshold (usuallyenergy or loudness threshold) for each frequency band to determine if itneeds to be enhanced, that is, those frequency bands above respectiveenergy/loudness thresholds will be enhanced. For adjusting thethresholds, the adjusting unit 300A may be configured to positivelycorrelate the thresholds with a confidence value of short-term musicand/or noise and/or background sounds, and/or negatively correlate thethresholds with a confidence value of speech. For example, thethresholds can be lowered down if speech confidence is high, assumingmore reliable speech detection, to allow more frequency bands to beenhanced; on the other hand, when music confidence value is high, thethresholds can be increased to make less frequency bands to be enhanced(and thus fewer artifacts).

2.3 Adjustment to Background Level

Another component in the dialog enhancer is minimum tracking unit 4022,as shown in FIG. 15, which is used to estimate the background level inthe audio signal (for SNR estimation, and frequency band thresholdestimation as mentioned in Section 2.2). It can be also tuned based onthe confidence values of audio content types. For example, if speechconfidence is high, the minimum tracking unit can be more confident toset the background level to the current minimum. If music confidence ishigh, the background level can be set to a little higher than thatcurrent minimum, or in another way, set to a weighted average of currentminimum and the energy of current frame, with a large weight on thecurrent minimum. If noise and background confidence is high, thebackground level can be set much higher than the current minimum value,or in another way, set to a weighted average of current minimum and theenergy of current frame, with a small weight on the current minimum.

Thus, the adjusting unit 300A may be configured to assign an adjustmentto the background level estimated by the minimum tracking unit, whereinthe adjusting unit is further configured to positively correlate theadjustment with a confidence value of short-term music and/or noiseand/or background sound, and/or negatively correlate the adjustment witha confidence value of speech. In a variant, the adjusting unit 300A maybe configured to correlate the adjustment with the confidence value ofnoise and/or background more positively than the short-term music.

2.4 Combination of Embodiments and Application Scenarios

Similar to Part 1, all the embodiments and variants there of discussedabove may be implemented in any combination thereof, and any componentsmentioned in different parts/embodiments but having the same or similarfunctions may be implemented as the same or separate components.

For example, any two or more of the solutions described in Sections 2.1to 2.3 may be combined with each other. And these combinations may befurther combined with any embodiment described or implied in Part 1 andthe other parts that will be described later. Especially, many formulasare actually applicable to each kind of audio improving device ormethod, but they are not necessarily recited or discussed in each partof this disclosure. In such a situation, cross-reference may be madeamong the parts of this disclosure for applying a specific formuladiscussed in one part to another part, with only relevant parameter(s),coefficient(s), power(s)(exponents) and weight(s) being adjustedproperly according to specific requirements of the specific application.

2.5 Dialog Enhancer Controlling Method

Similar to Part 1, in the process of describing the dialog enhancercontroller in the embodiments hereinbefore, apparently disclosed arealso some processes or methods. Hereinafter a summary of these methodsis given without repeating some of the details already discussedhereinbefore.

Firstly, the embodiments of the audio processing method as discussed inPart 1 may be used for a dialog enhancer, the parameter(s) of which isone of the targets to be adjusted by the audio processing method. Fromthis point of view, the audio processing method is also a dialogenhancer controlling method.

In this section, only those aspects specific to the control of thedialog enhancer will be discussed. For general aspects of thecontrolling method, reference may be made to Part 1.

According to one embodiment, the audio processing method may furthercomprise dialog enhancement processing, and the operation of adjusting1104 comprises positively correlating the level of dialog enhancementwith the confidence value of movie-like media and/or VoIP, and ornegatively correlating the level of dialog enhancement with theconfidence value of the long-term music and/or game. That is, dialogenhancement is mainly directed to the audio signal in the context ofmovie-like media, or VoIP.

More specifically, the operation of adjusting 1104 may comprisepositively correlating the level of dialog enhancement of the dialogenhancer with the confidence value of speech.

The present application may also adjust frequency bands to be enhancedin the dialog enhancement processing. As shown in FIG. 16, thethresholds (usually energy or loudness) for determining whetherrespective frequency bands are to be enhanced may be adjusted based onthe confidence value(s) of identified audio types (operation 1602)according to the present application. Then, within the dialog enhancer,based on the adjusted thresholds, frequency bands above respectivethresholds are selected (operation 1604) and enhanced (operation 1606).

Specifically, the operation of adjusting 1104 may comprise positivelycorrelating the thresholds with a confidence value of short-term musicand/or noise and/or background sounds, and/or negatively correlating thethresholds with a confidence value of speech.

The audio processing method (especially the dialog enhancementprocessing) generally further comprises estimating the background levelin the audio signal, which is generally implemented by a minimumtracking unit 4022 realized in the dialog enhancer 402, and used in SNRestimation or frequency band threshold estimation. The presentapplication may also be used to adjust the background level. In such asituation, after the background level is estimated (operation 1702), itis first adjusted based on the confidence value(s) of the audio type(s)(operation 1704), and then is used in the SNR estimation and/orfrequency band threshold estimation (operation 1706). Specifically, theoperation of adjusting 1104 may be configured to assign an adjustment tothe estimated background level, wherein the operation of adjusting 1104may be further configured to positively correlate the adjustment with aconfidence value of short-term music and/or noise and/or backgroundsound, and/or negatively correlate the adjustment with a confidencevalue of speech.

More specifically, the operation of adjusting 1104 may be configured tocorrelate the adjustment with the confidence value of noise and/orbackground more positively than the short-term music.

Similar to the embodiments of the audio processing apparatus, anycombination of the embodiments of the audio processing method and theirvariations are practical on one hand; and on the other hand, everyaspect of the embodiments of the audio processing method and theirvariations may be separate solutions. In addition, any two or moresolutions described in this section may be combined with each other, andthese combinations may be further combined with any embodiment describedor implied in Part 1 and the other parts that will be described later.

Part 3: Surround Virtualizer Controller and Controlling Method

A surround virtualizer enables a surround sound signal (such asmultichannel 5.1 and 7.1) to be rendered over the internal speakers ofthe PC or over headphones. That is, with stereo devices such as internallaptop speakers or headphones, it creates virtually surround effect andprovides cinematic experience for consumers. Head Related TransferFunctions (HRTFs) are usually utilized in the surround virtualizer tosimulate the arrival of sound at the ears coming from the variousspeaker locations associated with the multi-channel audio signal.

While current surround virtualizer works well on headphones, it worksdifferently on different contents with built-in speakers. In general,movie-like media content enables Surround Virtualizer for speakers,while music does not since it may sound too thin.

Since the same parameters in Surround Virtualizer cannot create goodsound image for both movie-like media and music content simultaneously,parameters have to be tuned based on the content more precisely. Withaudio type information available, especially the music confidence valueand speech confidence value, as well as some other content typeinformation and context information, the work can be done with thepresent application.

Similar to Part 2, as a specific example of the audio processingapparatus and method discussed in Part 1, the surround virtualizer 404may make use of all the embodiments discussed in Part 1 and anycombinations of those embodiments disclosed therein. Specifically, inthe case of controlling the surround virtualizer 404, the audioclassifier 200 and the adjusting unit 300 in the audio processingapparatus 100 as shown in FIGS. 1-10 may constitute a surroundvirtualizer controller 1800 as shown in FIG. 18. In this embodiment,since the adjusting unit is specific to the surround virtualizer 404, itmay be referred to as 300B. And, similar to Part 2, the audio classifier200 may comprise at least one of the audio content classifier 202 andthe audio context classifier 204, and the surround virtualizercontroller 1800 may further comprise at least one of the type smoothingunit 712, the parameter smoothing unit 814 and the timer 916.

Therefore, in this part, we will not repeat those contents alreadydescribed in Part 1, and just give some specific examples thereof.

For a surround virtualizer, the adjustable parameters include but arenot limited to surround boost amount and the start frequency for thesurround virtualizer 404.

3.1 Surround Boost Amount

When involving the surround boost amount, the adjusting unit 300B may beconfigured to positively correlate the surround boost amount of thesurround virtualizer 404 with a confidence value of noise and/orbackground and/or speech, and/or negatively correlate the surround boostamount with a confidence value of short-term music.

Specifically, to modify the surround virtualizer 404 in order that music(content type) sounds acceptable, an example implementation of theadjusting unit 300B could tune the amount of the surround boost based onthe short-term music confidence value, such as:SB∝(1−Conf_(music))  (5)where SB indicates surround boost amount, Conf_(music) is the confidencevalue of short-term music.

It helps to decrease the surround boost for music and prevent it fromsounding washed out.

Similarly, the speech confidence value can be also utilized, forexample:SB∝(1−Conf_(music))*Conf_(speech)α  (6)where Conf_(speech) is the confidence value of speech, α is a weightingcoefficient in the form of exponent, and may be in the range of 1-2.This formula indicates the surround boost amount will be high for onlypure speech (high speech confidence and low music confidence).

Or we can consider only the confidence value of speech:SB∝Conf_(speech)  (7)

Various variants can be designed in a similar way. Especially, for noiseor background sound, formulae similar to formulas (5) to (7) may beconstructed. In addition, the effects of the four content types may beconsidered together in any combination. In such a situation, noise andbackground are ambience sounds and they are more safe to have a largeboost amount; speech can have a middle boost amount, supposing a talkerusually sit in the front of the screen; and music uses less boostamount. Therefore, the adjusting unit 300B may be configured tocorrelate the surround boost amount with the confidence value of noiseand/or background more positively than the content type speech.

Supposing we predefined an expected boost amount (that is equivalent toa weight) for each content type, another alternative can be alsoapplied:

$\begin{matrix}{\hat{a} = \frac{\begin{matrix}{{a_{speech} \cdot {Conf}_{speech}} + {a_{music} \cdot {Conf}_{music}} +} \\{{a_{noise} \cdot {Conf}_{noise}} + {a_{bkg} \cdot {Conf}_{bkg}}}\end{matrix}}{{Conf}_{speech} + {Conf}_{music} + {Conf}_{noise} + {Conf}_{bkg}}} & (8)\end{matrix}$where â is the estimated boost amount, the α with an subscript of thecontent type is the expected/predefined boost amount (weight) of thecontent type, Conf with an subscript of the content type is theconfidence value of the content type (where bkg represents “backgroundsound”). Depending on situations, a_(music) may be (but not necessarily)set as 0, indicating that the surround virtualizer 404 will be disabledfor pure music (content type).

From another point of view, the α with an subscript of the content typein formula (8) is the expected/predefined boost amount of the contenttype, and the quotient of the confidence value of the correspondingcontent type divided by the sum of the confidence values of all theidentified content types may be regarded as normalized weight of thepredefined/expected boost amount of the corresponding content type. Thatis, the adjusting unit 300B may be configured to consider at least someof the multiple content types through weighting the predefined boostamounts of the multiple content types based on the confidence values.

As to the context type, the adjusting unit 300B may be configured topositively correlate the surround boost amount of the surroundvirtualizer 404 with a confidence value of movie-like media and/or game,and/or negatively correlate the surround boost amount with a confidencevalue of long-term music and/or VoIP. Then, formulas similar to (5) to(8) may be constructed.

As a special example, the surround virtualizer 404 can be enabled forpure movie-like media and/or game, but disabled for music and/or VoIP.Meanwhile, the boost amount of the surround virtualizer 404 can be setdifferently for movie-like media and game. movie-like media uses ahigher boost amount, and game uses less. Therefore, the adjusting unit300B may be configured to correlate the surround boost amount with theconfidence value of movie-like media more positively than game.

Similar to the content type, the boost amount of an audio signal canalso be set to a weighted average of the confidence values of thecontext types:

$\begin{matrix}{\hat{a} = \frac{\begin{matrix}{{a_{MOVE} \cdot {Conf}_{MOVIE}} + {a_{MUSIC} \cdot {Conf}_{MUSIC}} +} \\{{a_{GAME} \cdot {Conf}_{GAME}} + {a_{VOIP} \cdot {Conf}_{VOIP}}}\end{matrix}}{{Conf}_{MOVIE} + {Conf}_{MUSIC} + {Conf}_{GAME} + {Conf}_{VOIP}}} & (9)\end{matrix}$where â is the estimated boost amount, the α with an subscript of thecontext type is the expected/predefined boost amount (weight) of thecontext type, Conf with an subscript of the context type is theconfidence value of the context type. Depending on situations, a_(MUSIC)and a_(VOIP) may be (but not necessarily) set as 0, indicating that thesurround virtualizer 404 will be disabled for pure music (context type)and or pure VoIP.

Again, similar to the content type, the α with an subscript of thecontext type in formula (9) is the expected/predefined boost amount ofthe context type, and the quotient of the confidence value of thecorresponding context type divided by the sum of the confidence valuesof all the identified context types may be regarded as normalized weightof the predefined/expected boost amount of the corresponding contexttype. That is, the adjusting unit 300B may be configured to consider atleast some of the multiple context types through weighting thepredefined boost amounts of the multiple context types based on theconfidence values.

3.2 Start Frequency

Other parameters can be also modified in the surround virtualizer, suchas the start frequency. Generally, high frequency components in an audiosignal are more suitable to be spatially rendered. For example, inmusic, it will sound strange if the bass is spatially rendered to havemore surround effects. Therefore, for a specific audio signal, thesurround virtualizer need determine a frequency threshold, thecomponents above which are spatially rendered while the components belowwhich are retained. The frequency threshold is the start frequency.

According to an embodiment of the present application, the startfrequency for the surround virtualizer can be increased on the musiccontent so that more bass can be retained for music signals. Then, theadjusting unit 300B may be configured to positively correlate the startfrequency of the surround virtualizer with a confidence value ofshort-term music.

3.3 Combination of Embodiments and Application Scenarios

Similar to Part 1, all the embodiments and variants thereof discussedabove may be implemented in any combination thereof, and any componentsmentioned in different parts/embodiments but having the same or similarfunctions may be implemented as the same or separate components.

For example, any two or more of the solutions described in Sections 3.1and 3.2 may be combined with each other. And any of the combinations maybe further combined with any embodiment described or implied in Part 1,Part 2 and the other parts that will be described later.

3.4 Surround Virtualizer Controlling Method

Similar to Part 1, in the process of describing the surround virtualizercontroller in the embodiments hereinbefore, apparently disclosed arealso some processes or methods. Hereinafter a summary of these methodsis given without repeating some of the details already discussedhereinbefore.

Firstly, the embodiments of the audio processing method as discussed inPart 1 may be used for a surround virtualizer, the parameter(s) of whichis one of the targets to be adjusted by the audio processing method.From this point of view, the audio processing method is also a surroundvirtualizer controlling method.

In this section, only those aspects specific to the control of thesurround virtualizer will be discussed. For general aspects of thecontrolling method, reference may be made to Part 1.

According to one embodiment, the audio processing method may furthercomprise surround virtualizing processing, and the operation ofadjusting 1104 may be configured to positively correlate the surroundboost amount of the surround virtualizing processing with a confidencevalue of noise and/or background and/or speech, and/or negativelycorrelate the surround boost amount with a confidence value ofshort-term music.

Specifically, the operation of adjusting 1104 may be configured tocorrelate the surround boost amount with the confidence value of noiseand/or background more positively than the content type speech.

Alternatively or additionally, the surround boost amount may also beadjusted based on the confidence value(s) of context type(s).Specifically, the operation of adjusting 1104 may be configured topositively correlate the surround boost amount of the surroundvirtualizing processing with a confidence value of movie-like mediaand/or game, and/or negatively correlate the surround boost amount witha confidence value of long-term music and/or VoIP.

More specifically, the operation of adjusting 1104 may be configured tocorrelate the surround boost amount with the confidence value ofmovie-like media more positively than game.

Another parameter to be adjusted is the start frequency for the surroundvirtualizing processing. As shown in FIG. 19, the start frequency isadjusted firstly based on the confidence value(s) of the audio type(s)(operation 1902), then the surround virtualizer process those audiocomponents above the start frequency (operation 1904). Specifically, theoperation of adjusting 1104 may be configured to positively correlatethe start frequency of the surround virtualizing processing with aconfidence value of short-term music.

Similar to the embodiments of the audio processing apparatus, anycombination of the embodiments of the audio processing method and theirvariations are practical on one hand; and on the other hand, everyaspect of the embodiments of the audio processing method and theirvariations may be separate solutions. In addition, any two or moresolutions described in this section may be combined with each other, andthese combinations may be further combined with any embodiment describedor implied in the other parts of this disclosure.

Part 4: Volume Leveler Controller and Controlling Method

The volume of different audio sources or different pieces in the sameaudio source sometime changes a lot. It is annoying since users have toadjust the volume frequently. Volume Leveler (VL) aims at tuning thevolume of the audio content on playback and keeping it almost consistentover the timeline based on a target loudness value. Example VolumeLevelers are presented in A. J. Seefeldt et al. “Calculating andAdjusting the Perceived Loudness and/or the Perceived Spectral Balanceof an Audio Signal”, published as US2009/0097676A1; B. G. Grockett etal. “Audio Gain Control Using Specific-Loudness-Based Auditory EventDetection”, published as WO2007/127023A1; and A. Seefeldt et al. “AudioProcessing Using Auditory Scene Analysis and Spectral Skewness”,published as WO 2009/011827 A1. The three documents are incorporatedherein in their entirety by reference.

The volume leveler continuously measures the loudness of an audio signalin some manner and then modifies the signal by an amount of gain that isa scaling factor to modify the loudness of an audio signal and usuallyis a function of the measured loudness, the desired target loudness, andseveral other factors. A number of factors needed to be considered toestimate a proper gain, with underlying criteria to both approach to thetarget loudness and keep the dynamic range. It usually comprises severalsub-elements such as automatic gain control (AGC), auditory eventdetection, dynamic range control (DRC).

A control signal is generally applied in the volume leveler to controlthe “gain” of the audio signal. For example, a control signal can be anindicator of the change in the magnitude of the audio signal derived bypure signal analysis. It can be also an audio event indicator torepresent if a new audio event appears, through psycho-acousticalanalysis such as auditory scene analysis or specific-loudness-basedauditory event detection. Such a control signal is applied in volumeleveler for gain controlling, for example, by ensuring that the gain isnearly constant within an auditory event and by confining much of thegain change to the neighborhood of an event boundary, in order to reducepossible audible artifacts due to a rapid change of the gain in theaudio signal.

However, the conventional methods of deriving control signals cannotdifferentiate informative auditory events from non-informative(interfering) auditory events. Here, the informative auditory eventstands for the audio event that contains meaningful information and maybe paid more attention by users, such as dialog and music, while thenon-informative signal does not contain meaningful information forusers, such as noise in VoIP. As a consequent, the non-informativesignals may also be applied by a large gain and boosted to close to thetarget loudness. It will be unpleasing in some applications. Forexample, in VoIP calls, the noise signal that appears in the pause of aconversation is often boosted up to a loud volume after processed by avolume leveler. This is unwanted by users.

In order to address this problem at least in part, the presentapplication proposes to control the volume leveler based on theembodiments discussed in Part 1.

Similar to Part 2 and Part 3, as a specific example of the audioprocessing apparatus and method discussed in Part 1, the volume leveler406 may make use of all the embodiments discussed in Part 1 and anycombinations of those embodiments disclosed therein. Specifically, inthe case of controlling the volume leveler 406, the audio classifier 200and the adjusting unit 300 in the audio processing apparatus 100 asshown in FIGS. 1-10 may constitute a volume leveler 406 controller 2000as shown in FIG. 20. In this embodiment, since the adjusting unit isspecific to the volume leveler 406, it may be referred to as 300C.

That is, based on the disclosure of Part 1, a volume leveler controller2000 may comprise an audio classifier 200 for continuously identifyingthe audio type (such as content type and/or context type) of an audiosignal; and an adjusting unit 300C for adjusting a volume leveler in acontinuous manner based on the confidence value of the audio type asidentified. Similarly, the audio classifier 200 may comprise at leastone of the audio content classifier 202 and the audio context classifier204, and the volume leveler controller 2000 may further comprise atleast one of the type smoothing unit 712, the parameter smoothing unit814 and the timer 916.

Therefore, in this part, we will not repeat those contents alreadydescribed in Part 1, and just give some specific examples thereof.

Different parameters in the volume leveler 406 can be adaptively tunedbased on the classification results. We can tune the parameters directlyrelated to the dynamic gain or the range of the dynamic gain, forexample, by reducing the gain for the non-informative signals. We canalso tune the parameters which indicate the degree of the signal being anew perceptible audio event, and then indirectly control the dynamicgain (the gain will change slowly within an audio event, but may changerapidly at the boundary of two audio events). In this application,several embodiments of the parameter tuning or volume levelercontrolling mechanism are presented.

4.1 Informative and Interfering Content Types

As mentioned above, in connection with the control of the volumeleveler, the audio content types may be classified as informativecontent types and interfering content types. And the adjusting unit 300Cmay be configured to positively correlate the dynamic gain of the volumeleveler with informative content types of the audio signal, andnegatively correlate the dynamic gain of the volume leveler withinterfering content types of the audio signal.

As an example, supposing noise is interfering (non-informative) and itwill be annoying to be boosted into a loud volume, the parameterdirectly controlling the dynamic gain, or the parameter indicating newaudio events can be set to be proportional to a decreasing function ofthe noise confidence value (Conf_(noise)), such asGainControl∝1−Conf_(noise)  (10)

Here, for simplicity, we use the symbol GainControl to represent all theparameters (or their effects) related to gain controlling in the volumeleveler, since different implementations of volume leveler may usedifferent names of parameters with different underlying meaning. Usingthe single term GainControl can have a short expression without losinggenerality. In essence, adjusting these parameters is equivalent toapplying a weight on the original gain, either linear or non-linear. Asone example, the GainControl can be directly used to scale the gain sothat the gain will be small if GainControl is small. As another specificexample, the gain is indirectly controlled by scaling with GainControlthe event control signal described in B. G. Grockett et al. “Audio GainControl Using Specific-Loudness-Based Auditory Event Detection”,published as WO2007/127023A1, which is incorporated herein in itsentirety by reference. In this case, when GainControl is small, thecontrols of the volume leveler's gain are modified to prevent the gainfrom changing significantly with time. When GainControl is high, thecontrols are modified so that the gain of the leveler is allowed tochange more freely.

With the gain control described in formula (10) (either directly scalingthe original gain or the event control signal), the dynamic gain of anaudio signal is correlated (linear or nonlinearly) to its noiseconfidence value. If the signal is noise with a high confidence value,the final gain will be small due to the factor (1−Conf_(noise)). In thisway, it avoids boosting a noise signal into an unpleasing loud volume.

As an example variant from formula (10), if the background sound is alsonot interested in an application (such as in VoIP), it can be dealt withsimilarly and applied by a small gain as well. A controlling functioncan consider both noise confidence value (Conf_(noise)) and backgroundconfidence value (Conf_(bkg)), for exampleGainControl∝(1−Conf_(noise))·(1−Conf_(bkg))  (11)In above formula, since both noise and background sounds are not wanted,the GainControl is equally affected by the confidence value of noise andthe confidence value of background, and it may be regarded that noiseand background sounds have the same weight. Depending on situations,they may have different weights. For example, we may give the confidencevalues of noise and background sounds (or their difference with 1)different coefficients or different exponents (α and γ). That is,formula (11) may be rewritten as:GainControl∝(1−Conf_(noise))^(α)·(1−Conf_(bkg))^(γ)  (12)orGainControl∝(1−Conf_(noise) ^(α))·(1−Conf_(bkg) ^(γ))  (13)

Alternatively, the adjusting unit 300C may be configured to consider atleast one dominant content type based on the confidence values. Forexample:GainControl∝1−max(Conf_(noise),Conf_(bkg))  (14)Both formula (11) (and its variants) and formula (14) indicate a smallgain for noise signals and background sound signals, and the originalbehavior of the volume leveler is kept only when both noise confidenceand background confidence is small (such as in speech and music signal)so that GainControl is close to one.

The example above is to consider the dominant interfering content type.Depending on situation, the adjusting unit 300C may also be configuredto consider the dominant informative content type based on theconfidence values. To be more general, the adjusting unit 300C may beconfigured to consider at least one dominant content type based on theconfidence values, no matter whether the identified audio typesare/include informative and/or interfering audio types.

As another example variant of formula (10), supposing speech signal isthe most informative content and needs less modification on the defaultbehavior of the volume leveler, the controlling function can considerboth noise confidence value (Conf_(noise)) and speech confidence value(Conf_(speech)), asGainControl∝1−Conf_(noise)·(1−Conf_(speech))  (15)With this function, a small GainControl is obtained only for thosesignals with high noise confidence and low speech confidence (e.g., purenoise), and the GainControl will be close to 1 if the speech confidenceis high (and thus keep the original behavior of the volume leveler).More generally, it can be regarded that the weight of one content type(such as Conf_(noise)) may be modified with the confidence value of atleast one other content type (such as Conf_(speech)). In above formula(15), it can be regarded that the confidence of speech changes theweight coefficient of the confidence of noise (another kind of weight ifcompared to the weights in formula (12 and 13)). In other words, informula (10), the coefficient of Conf_(noise) can be regarded as 1;while in formula (15), some other audio types (such as speech, but notlimited thereto) will affect the importance of the confidence value ofnoise, thus we can say the weight of Conf_(noise) is modified by theconfidence value of the speech. In the context of the presentdisclosure, the term “weight” shall be construed to include this. Thatis, it indicates the importance of a value, but not necessarilynormalized. Reference may be made to Section 1.4.

From another viewpoint, similar to formula (12) and (13), weights in theform of exponents can be applied on the confidence values in the abovefunction to indicate the priority (or importance) of different audiosignals, for example, the formula (15) can be changed to:GainControl∝1−Conf_(noise) ^(α)·(1−Conf_(speech))^(γ)  (16)where α and γ are two weights, which can be set smaller if it isexpected to be more respondent to modify leveler parameters.

The formulas (10)-(16) can be freely combined to form variouscontrolling functions which may be suitable in different applications.The confidence values of other audio content types, such as musicconfidence value, can be also easily incorporated in the controllingfunctions in a similar way.

In the case where the GainControl is used to tune the parameters whichindicate the degree of the signal being a new perceptible audio event,and then indirectly control the dynamic gain (the gain will changeslowly within an audio event, but may change rapidly at the boundary oftwo audio events), it may be regarded that there is another transferfunction between the confidence value of content types and the finaldynamic gain.

4.2 Content Types in Different Contexts

The above controlling functions in formula (10)-(16) take considerationof the confidence values of audio content types, such as noise,background sounds, short-term music, and speech, but do not considertheir audio contexts where the sounds come from, such as movie-likemedia and VoIP. It is possible that the same audio content type mightneed to be processed differently in different audio contexts, forexample, the background sounds. Background sound comprises varioussounds such as car engine, explosion, and applause. It may be notmeaningful in a VoIP call but it could be important in a movie-likemedia. This indicates that the interested audio contexts need to beidentified and different controlling functions need to be designed fordifferent audio contexts.

Therefore, the adjusting unit 300C may be configured to regard thecontent type of the audio signal as informative or interfering based onthe context type of the audio signal. For example, by considering noiseconfidence value and background confidence value, and differentiatingVoIP and non-VoIP contexts, an audio context-dependent controllingfunction can be,

-   -   if the audio context is VoIP        GainControl∝1−max(Conf_(noise),Conf_(bkg))    -   else        GainControl∝1−Conf_(noise)  (17)        That is, in the VoIP context, noise and background sounds are        regarded as interfering content types; while in the non-VoIP        context, background sounds is regarded as informative content        type.

As another example, an audio context-dependent controlling functionconsidering confidence values of speech, noise, and background, anddifferentiating VoIP and non-VoIP contexts, could be

-   -   if the audio context is VoIP        GainControl∝1−max(Conf_(noise),Conf_(bkg))    -   else        GainControl∝1−Conf_(noise)·(1−Conf_(speech))  (18)        Here, speech is emphasized as an informative content type.

Supposing music is also important informative information in non-VoIPcontext, we can extend the second part of formula (18) to:GainControl∝1−Conf_(noise)·(1−max(Conf_(speech),Conf_(music)))  (19)

In fact, each of the controlling functions in (10)-(16) or theirvariants can be applied in different/corresponding audio contexts. Thus,it can generate a large number of combinations to form audiocontext-dependent controlling functions.

Besides VoIP and non-VoIP contexts as differentiated and utilized informula (17) and (18), other audio contexts, such as movie-like media,long-term music, and game, or low-quality audio and high-quality audio,can be utilized in a similar way.

4.3 Context Types

Context types can also be directly used to control the volume leveler toavoid those annoying sounds, such as noise, to be boosted too much. Forexample, the VoIP confidence value can be used to steer the volumeleveler, making it less sensitive when its confidence value is high.

Specifically, with the VoIP confidence value Conf_(VOIP), the level ofthe volume leveler can be set to be proportional to (1−Conf_(VOIP)).That is, the volume leveler is almost deactivated in the VoIP content(when the VoIP confidence value is high), which is consistent with thetraditional manual setup (preset) that disables the volume leveler forVoIP context.

Alternatively, we can set different dynamic gain ranges for differentcontexts of audio signals. In general, a VL (volume leveler) amountfurther adjusts the amount of gain applied on an audio signal, and canbe seen as another (nonlinear) weight on the gain. In one embodiment, asetup could be:

TABLE 1 MOVIE-LIKE LONG-TERM MEDIA MUSIC VOIP GAME VL Amount high middleOff low (or lowest)

Furthermore, supposing an expected VL amount is predefined for eachcontext type. For example, the VL amount is set as 1 for Movie-likemedia, 0 for VoIP, 0.6 for Music, and 0.3 for Game, but the presentapplication is not limited thereto. According to the example, if therange of the dynamic gain of movie-like media is 100%, then the range ofthe dynamic gain of VoIP is 60%, and so on. If the classification of theaudio classifier 200 is based on hard decision, then the range of thedynamic gain may be directly set as the above example. If theclassification of the audio classifier 200 is based on soft decision,then the range may be adjusted based on the confidence value of thecontext type.

Similarly, the audio classifier 200 may identify multiple context typesfrom the audio signal, and the adjusting unit 300C may be configured toadjust the range of the dynamic gain by weighting the confidence valuesof the multiple content types based on the importance of the multiplecontent types.

Generally, for context type, the functions similar to (10)-(16) can bealso used here to set the appropriate VL amount adaptively, with thecontent types therein replaced with context types, and actually Table 1reflects the importance of a different context types.

From another point of view, the confidence value may be used to derive anormalized weight as discussed in Section 1.4. Supposing a specificamount is predefined for each context type in Table 1, then a formulasimilar to formula (9) can be also applied. Incidentally, similarsolutions may also be applied to multiple content types and any otheraudio types.

4.4 Combination of Embodiments and Application Scenarios

Similar to Part 1, all the embodiments and variants there of discussedabove may be implemented in any combination thereof, and any componentsmentioned in different parts/embodiments but having the same or similarfunctions may be implemented as the same or separate components. Forexample, any two or more of the solutions described in Sections 4.1 to4.3 may be combined with each other. And any of the combinations may befurther combined with any embodiment described or implied in Parts 1-3and the other parts that will be described later.

FIG. 21 illustrates the effect of the volume leveler controller proposedin the application by comparing an original short-term segment (FIG.21(A)), the short-term segment processed by a conventional volumeleveler without parameter modification (FIG. 21(B), and the short-termsegment processed by a volume leveler as presented in this application(FIG. 21(C)). As seen, in the conventional volume leveler as shown inFIG. 21(B), the volume of the noise (the second half of the audiosignal) is also boosted and is annoying. In contrast, in the new volumeleveler as shown in FIG. 21(C), the volume of the effective part of theaudio signal is boosted without apparently boosting the volume of thenoise, giving the audience good experience.

4.5 Volume Leveler Controlling Method

Similar to Part 1, in the process of describing the volume levelercontroller in the embodiments hereinbefore, apparently disclosed arealso some processes or methods. Hereinafter a summary of these methodsis given without repeating some of the details already discussedhereinbefore.

Firstly, the embodiments of the audio processing method as discussed inPart 1 may be used for a volume leveler, the parameter(s) of which isone of the targets to be adjusted by the audio processing method. Fromthis point of view, the audio processing method is also A volume levelercontrolling method.

In this section, only those aspects specific to the control of thevolume leveler will be discussed. For general aspects of the controllingmethod, reference may be made to Part 1.

According to the present application, A volume leveler controllingmethod is provided, comprising identifying the content type of an audiosignal in real time, and adjusting a volume leveler in a continuousmanner based on the content type as identified, by positivelycorrelating the dynamic gain of the volume leveler with informativecontent types of the audio signal, and negatively correlating thedynamic gain of the volume leveler with interfering content types of theaudio signal.

The content type may comprise speech, short-term music, noise andbackground sound. Generally, noise is regarded as an interfering contenttype.

When adjusting the dynamic gain of the volume leveler, it may beadjusted directly based on the confidence value of the content type, ormay be adjusted via a transfer function of the confidence value of thecontent type.

As already described, the audio signal may be classified into multipleaudio types at the same time. When involving multiple content types, theadjusting operation 1104 may be configured to consider at least some ofthe multiple audio content types through weighting the confidence valuesof the multiple content types based on the importance of the multiplecontent types, or through weighting the effects of the multiple contenttypes based on the confidence values. Specifically, and the adjustingoperation 1104 may be configured to consider at least one dominantcontent type based on the confidence values. When the audio signalcontains both interfering content type(s) and informative contenttype(s), the adjusting operation may be configured to consider at leastone dominant interfering content type based on the confidence values,and/or consider at least one dominant informative content type based onthe confidence values.

Different audio types may affect each other. Therefore, the adjustingoperation 1104 may be configured to modify the weight of one contenttype with the confidence value of at least one other content type.

As described in Part 1, the confidence value of the audio type of theaudio signal may be smoothed. For the detail of the smoothing operation,please refer to Part 1.

The method may further comprise identifying the context type of theaudio signal, wherein the adjusting operation 1104 may be configured toadjust the range of the dynamic gain based on the confidence value ofthe context type.

The role of a content type is limited by the context type where it islocated. Therefore, when both content type information and context typeinformation are obtained for an audio signal at the same time (that isfor the same audio segment), the content type of the audio signal may bedetermined as informative or interfering based on the context type ofthe audio signal. Further, the content type in an audio signal of adifferent context type may be assigned a different weight depending onthe context type of the audio signal. From another point of view, we canuse different weight (larger or smaller, plus value or minus value) toreflect the informative nature or interfering nature of a content type.

The context type of the audio signal may comprise VoIP, movie-likemedia, long-term music and game. And in the audio signal of the contexttype VoIP, the background sound is regarded as an interfering contenttype; while in the audio signal of the context type non-VoIP, thebackground and/or speech and/or music is regarded as an informativecontent type. Other context types may include high-quality audio orlow-quality audio.

Similar to the multiple content types, when the audio signal isclassified into multiple context types with corresponding confidencevalues at the same time (with respect to the same audio segment), theadjusting operation 1104 may be configured to consider at least some ofthe multiple context types through weighting the confidence values ofthe multiple context types based on the importance of the multiplecontext types, or through weighting the effects of the multiple contexttypes based on the confidence values. Specifically, the adjustingoperation may be configured to consider at least one dominant contexttype based on the confidence values.

Finally, the embodiments of the method as discussed in this section mayuse the audio classifying method as will be discussed in Parts 6 and 7,and detailed description is omitted here.

Similar to the embodiments of the audio processing apparatus, anycombination of the embodiments of the audio processing method and theirvariations are practical on one hand; and on the other hand, everyaspect of the embodiments of the audio processing method and theirvariations may be separate solutions. In addition, any two or moresolutions described in this section may be combined with each other, andthese combinations may be further combined with any embodiment describedor implied in the other parts of this disclosure.

Part 5: Equalizer Controller and Controlling Method

Equalization is usually applied on a music signal to adjust or modifyits spectral balance, as known as “tone” or “timbre”. A traditionalequalizer allows users to configure the overall profile (curve or shape)of the frequency response (gain) on each individual frequency band, inorder to emphasize certain instruments or remove undesired sounds.Popular music players, such as windows media player, usually provide agraphic equalizer to adjust the gain at each frequency band, and alsoprovide a set of equalizer presets for different music genres, such asRock, Rap, Jazz and Folk, to get best experience in listening todifferent genres of music. Once a preset is selected, or a profile isset, the same equalization gains will be applied on the signal, untilthe profile is modified manually.

In contrast, a dynamic equalizer provides a way to automatically adjustthe equalization gains at each frequency bands in order to keep overallconsistency of spectral balance with regard to a desired timbre or tone.This consistency is achieved by continuously monitoring the spectralbalance of the audio, comparing it to a desired preset spectral balance,and dynamically adjusting the applied equalization gains to transformthe audio's original spectral balance into the desired spectral balance.The desired spectral balance is manually selected or pre-set beforeprocessing.

Both kinds of the equalizers share the following disadvantage: the bestequalization profile, the desired spectral balance, or the relatedparameters have to be selected manually, and they cannot beautomatically modified based on the audio content on playback.Discriminating audio content types will be very important to provideoverall good quality for different kinds of audio signals. For example,different music pieces need different equalization profiles, such asthose of different genres.

In an equalizer system in which any kinds of audio signals (not justmusic) are possible to be input, the equalizer parameters need to beadjusted based on the content types. For example, equalizer is usuallyenabled on music signals, but disabled on speech signals, since it maychange the timbre of speech too much and correspondingly make the signalsound unnatural.

In order to address this problem at least in part, the presentapplication proposes to control the equalizer based on the embodimentsdiscussed in Part 1.

Similar to Parts 2-4, as a specific example of the audio processingapparatus and method discussed in Part 1, the equalizer 408 may make useof all the embodiments discussed in Part 1 and any combinations of thoseembodiments disclosed therein. Specifically, in the case of controllingthe equalizer 408, the audio classifier 200 and the adjusting unit 300in the audio processing apparatus 100 as shown in FIGS. 1-10 mayconstitute an equalizer 408 controller 2200 as shown in FIG. 22. In thisembodiment, since the adjusting unit is specific to the equalizer 408,it may be referred to as 300D.

That is, based on the disclosure of Part 1, an equalizer controller 2200may comprise an audio classifier 200 for continuously identifying theaudio type of an audio signal; and an adjusting unit 300D for adjustingan equalizer in a continuous manner based on the confidence value of theaudio type as identified. Similarly, the audio classifier 200 maycomprise at least one of the audio content classifier 202 and the audiocontext classifier 204, and the volume equalizer controller 2200 mayfurther comprise at least one of the type smoothing unit 712, theparameter smoothing unit 814 and the timer 916.

Therefore, in this part, we will not repeat those contents alreadydescribed in Part 1, and just give some specific examples thereof.

5.1 Control Based on Content Type

Generally speaking, for general audio content types such as music,speech, background sound and noise, the equalizer should be setdifferently on different content types. Similar to the traditionalsetup, the equalizer can be automatically enabled on music signals, butdisabled on speech; or in a more continuous manner, set a highequalization level on music signals and low equalization level on speechsignals. In this way, the equalization level of an equalizer canautomatically set for different audio content.

Specifically for music, it is observed that equalizer does not work sowell on a music piece that has a dominant source, since the timbre ofthe dominant source may change significantly and sound unnatural if aninappropriate equalization is applied. Considering this, it would bebetter to set a low equalization level on music pieces with dominantsources, while the equalization level can be kept high on music pieceswithout dominant sources. With this information, the equalizer canautomatically set the equalization level for different music content.

Music can also be grouped based on different properties, such as genre,instrument, and general music characteristics including rhythm, tempo,and timbre. In the same way that different equalizer presets are usedfor different music genres, these music groups/clusters may also havetheir own optimal equalization profiles or equalizer curves (intraditional equalizer) or optimal desired spectral balance (in dynamicequalizer).

As mentioned above, the equalizer is generally enabled on music contentbut disabled on speech since the equalizer may make dialog not sound sogood due to the timbre change. One method to automatically achieve it isto relate the equalization level to the content, in particular the musicconfidence value and/or speech confidence value obtained from the audiocontent classification module. Here, the equalization level can beexplained as the weight of the applied equalizer gains. The higher thelevel is, the stronger the applied equalization is. For the example, ifthe equalization level is 1, a full equalization profile gets applied;if the equalization level is zero, all gains are correspondingly 0 dBand thus non-equalization is applied. The equalization level may berepresented by different parameters in different implementations of theequalizer algorithms. An example embodiment of this parameter is theequalizer weight as implemented in A. Seefeldt et. al. “Calculating andAdjusting the Perceived Loudness and/or the Perceived Spectral Balanceof an Audio Signal”, published as US 2009/0097676 A1, which isincorporated herein in its entirety by reference.

Various controlling schemes can be designed to tune the equalizationlevel. For example, with the audio content type information, either thespeech confidence value or music confidence value can be used to set theequalization level, asL _(eq)∝Conf_(music)  (20)OrL _(eq)∝1−Conf_(speech)  (21)where L_(eq) is the equalization level and Conf_(music) Conf_(speech)stands for the confidence value of music and speech.

That is, the adjusting unit 300D may be configured to positivelycorrelate an equalization level with a confidence value of short-termmusic, or negatively correlate the equalization level with a confidencevalue of speech.

The speech confidence value and music confidence value can be furtherjointly used to set the equalization level. The general idea is that theequalization level should be high only when both music confidence valueis high and speech confidence value is low, and otherwise theequalization level is low. For example,L _(eq)=Conf_(music)(1−Conf_(speech) ^(α))  (22)where the speech confidence value is powered to α in order to deal withthe non-zero speech confidence in music signals which may frequentlyhappen. With the above formula, equalization will be fully applied (withthe level equal to 1) on the pure music signals without any speechcomponents. As stated in Part 1, α may be regarded as a weightingcoefficient based on the importance of the content type, and can betypically set to 1 to 2.

If posing greater weight on the confidence value of speech, theadjusting unit 300D may be configured to disable the equalizer 408 whenthe confidence value for the content type speech is greater than athreshold.

In above description the content types of music and speech are taken asexamples. Alternatively or additionally, the confidence values ofbackground sound and/or noise may also be considered. Specifically, theadjusting unit 300D may be configured to positively correlate anequalization level with a confidence value of background, and/ornegatively correlate the equalization level with a confidence value ofnoise.

As another embodiment, the confidence value may be used to derive anormalized weight as discussed in Section 1.4. Supposing an expectedequalization level is predefined for each content type (e.g., 1 formusic, 0 for speech, 0.5 for noise and background), a formula similar toformula (8) can be exactly applied.

The equalization level can be further smoothed to avoid rapid changewhich may introduce audible artifacts at transition points. This can bedone with the parameter smoothing unit 814 as described in Section 1.5.

5.2 Likelihood Of Dominant Sources In Music

In order to avoid the music with dominant sources being applied a highequalization level, the equalization level may be further correlated tothe confidence value Conf_(dom) indicating if a music piece contains adominant source, for example,L _(eq)=1−Conf_(dom)  (23)

In this way, the equalization level will be low on music pieces withdominant sources, and high on music pieces without dominant sources.

Here, although the confidence value of music with a dominant source isdescribed, we can also use the confidence value of music without adominant source. That is, the adjusting unit 300D may be configured topositively correlate an equalization level with a confidence value ofshort-term music without dominant sources, and/or negatively correlatethe equalization level with a confidence value of short-term music withdominant sources.

As stated in Section 1.1, although music and speech on one hand, andmusic with or without dominant sources on the other hand, are contenttypes on different hierarchical levels, they can be considered inparallel. By jointly considering the confidence value of dominantsources and the speech and music confidence values as described above,the equalization level can be set by combining at least one of formula(20)-(21) with (23). An example is combining all the three formula:L _(eq)=Conf_(music)(1−Conf_(speech))(1−Conf_(dom))  (24)

Different weights based on the importance of the content type can befurther applied to different confidence values for generality, such asin the manner of the formula (22).

As another example, supposing Conf_(dom) is computed only when the audiosignal is music, a stepwise function can be designed, as

$\begin{matrix}{L_{eq} = \{ \begin{matrix}( {1 - {Conf}_{dom}} ) & {{Conf}_{music} > {threshold}} \\{{Conf}_{music}( {1 - {conf}_{speech}^{\alpha}} )} & {otherwise}\end{matrix} } & (25)\end{matrix}$

This function sets the equalization level based on the confidence valueof dominant scores if the classification system fairly ascertain thatthe audio is music (the music confidence value is larger than athreshold); otherwise, it is set based on the music and speechconfidence values. That is, the adjusting unit 300D may be configured toconsider the short-term music without/with dominant sources when theconfidence value for the short-term music is greater than a threshold.Of course, either the first or the second half in formula (25) may bemodified in the manner of formula (20) to (24).

The same smoothing scheme as discussed in Section 1.5 can be applied aswell, and the time constant α can be further set based on the transitiontype, such as the transition from music with dominant sources to musicwithout dominant sources, or the transition from music without dominantsources to music with dominant sources. For this purpose, a similarformula as the formula (4′) can also be applied.

5.3 Equalizer Presets

Besides adaptively tuning the equalization level based on the confidencevalues of audio content types, appropriate equalization profiles ordesired spectral balance presets can also be automatically chosen fordifferent audio content, depending on their genre, instrument, or othercharacteristics. The music with the same genre, containing the sameinstrument, or having the same musical characteristics, can share thesame equalization profiles or desired spectral balance presets.

For generality, we use the term “music clusters” to represent the musicgroups with the same genre, the same instrument, or similar musicalattributes, and they can be regarded as another hierarchical level ofaudio content types as stated in Section 1.1. Appropriate equalizationprofile, equalization level, and/or desired spectral balance preset, maybe associated to each music cluster. The equalization profile is thegain curve applied on the music signal and can be any one of theequalizer presets used for different music genres (such as Classical,Rock, Jazz, and Folk), and the desired spectral balance presetrepresents the desired timbre for each cluster. FIG. 23 illustratesseveral examples of desired spectral balance presets as implemented inDolby Home Theater technologies. Each one describes the desired spectralshape across the audible frequency range. This shape is continuouslycompared to the spectral shape of the incoming audio, and equalizationgains are computed from this comparison to transform the spectral shapeof the incoming audio into that of the preset.

For a new music piece, the closest cluster can be determined (harddecision), or the confidence value with regard to each music cluster canbe computed (soft decision). Based on this information, properequalization profile, or desired spectral balance preset, can bedetermined for the given music piece. The simplest way is to assign itthe corresponding profile of the best matched cluster, asP _(eq) =P _(c*)  (26)where P_(eq) is the estimated equalization profile or desired spectralbalance preset, and c* is the index of the best matched music cluster(the dominant audio type), which can be obtained by picking up thecluster with the highest confidence value.

Moreover, there may be more than one music cluster having confidencevalue that is larger than zero, meaning the music piece has more or lesssimilar attributes as those clusters. For example, a music piece mayhave multiple instruments, or it may have attributes of multiple genres.It inspires another way to estimate the proper equalization profile byconsidering all the clusters, instead of by using only the closestcluster. For example, a weighted sum can be used:

$\begin{matrix}{P_{eq} = {\sum\limits_{c = 1}^{N}\;{w_{c}P_{c}}}} & (27)\end{matrix}$where N is the number of predefined clusters, and w_(c) is the weight ofthe designed profile P_(c) regarding each pre-defined music cluster(with index c), which should be normalized to 1 based on theircorresponding confidence values. In this way, the estimated profilewould be a mixture of the profiles of music clusters. For example, for amusic piece having both attributes of Jazz and Rock, the estimatedprofile would be something in between.

In some applications, we may not want to involve all the clusters asshown in formula (27). Only a subset of the clusters—the clusters mostrelated to the current music piece—need to be considered, the formula(27) can be slightly revised to:

$\begin{matrix}{P_{eq} = {\sum\limits_{c^{\prime} = 1}^{N^{\prime}}\;{w_{c^{\prime}}P_{c^{\prime}}}}} & (28)\end{matrix}$where the N′ is number of clusters to be considered, and c′ is thecluster index after decreasingly sorting the clusters based on theirconfidence values. By using a subset, we can focus more on the mostrelated clusters and exclude those less relevant. In other words, theadjusting unit 300D may be configured to consider at least one dominantaudio type based on the confidence values.

In the description above, music clusters are taken as example. In fact,the solutions are applicable to audio types on any hierarchical level asdiscussed in Section 1.1. Therefore, in general, the adjusting unit 300Dmay be configured to assign an equalization level and/or equalizationprofile and/or spectral balance preset to each audio type.

5.4 Control Based on Context Type

In the previous sections, discussion is focused on various contenttypes. In more embodiments to be discussed in this section, context typemay be alternatively or additionally considered.

In general, the equalizer is enabled for music but disabled formovie-like media content since equalizer may make dialogs in movie-likemedia not sound so good due to obvious timbre change. It indicates thatthe equalization level may be related to the confidence value of thelong-term music and/or the confidence value of movie-like media:L _(eq)∝Conf_(MUSIC)  (29)OrL _(eq)∝1−Conf_(MOVIE)  (30)where L_(eq) is the equalization level, Conf_(MUSIC) and Conf_(MOVIE)stands for the confidence value of long-term music and movie-like media.

That is, the adjusting unit 300D may be configured to positivelycorrelate an equalization level with a confidence value of long-termmusic, or negatively correlate the equalization level with a confidencevalue of movie-like media.

That is, for a movie-like media signal, the movie-like media confidencevalue is high (or music confidence is low), and thus the equalizationlevel is low; on the other hand, for a music signal, the movie-likemedia confidence value will be low (or music confidence is high) andthus the equalization level is high.

The solutions shown in formula (29) and (30) may be modified in the samemanner as formula (22) to (25), and/or may be combined with any one ofthe solutions shown in formula (22) to (25).

Additionally or alternatively, the adjusting unit 300D may be configuredto negatively correlate the equalization level with a confidence valueof game.

As another embodiment, the confidence value may be used to derive anormalized weight as discussed in Section 1.4. Supposing an expectedequalization level/profile is predefined for each context type(equalization profiles are shown in the following Table 2), a formulasimilar to formula (9) can be also applied.

TABLE 2 MOVIE-LIKE LONG-TERM MEDIA MUSIC VoIP GAME equalization Profile1 Profile 2 Profile 3 Profile 4 profile

Here, in some profiles, all the gains can be set to zero, as a way todisable the equalizer for that certain context type, such as movie-likemedia and game.

5.5 Combination of Embodiments and Application Scenarios

Similar to Part 1, all the embodiments and variants there of discussedabove may be implemented in any combination thereof, and any componentsmentioned in different parts/embodiments but having the same or similarfunctions may be implemented as the same or separate components.

For example, any two or more of the solutions described in Sections 5.1to 5.4 may be combined with each other. And any of the combinations maybe further combined with any embodiment described or implied in Parts1-4 and the other parts that will be described later.

5.6 Equalizer Controlling Method

Similar to Part 1, in the process of describing the equalizer controllerin the embodiments hereinbefore, apparently disclosed are also someprocesses or methods. Hereinafter a summary of these methods is givenwithout repeating some of the details already discussed hereinbefore.

Firstly, the embodiments of the audio processing method as discussed inPart 1 may be used for an equalizer, the parameter(s) of which is one ofthe targets to be adjusted by the audio processing method. From thispoint of view, the audio processing method is also an equalizercontrolling method.

In this section, only those aspects specific to the control of theequalizer will be discussed. For general aspects of the controllingmethod, reference may be made to Part 1.

According to embodiments, an equalizer controlling method may compriseidentifying the audio type of an audio signal in real time, andadjusting an equalizer in a continuous manner based on the confidencevalue of the audio type as identified.

Similar to other parts of the present application, when involvingmultiple audio types with corresponding confidence values, the operationof adjusting 1104 may be configured to consider at least some of themultiple audio types through weighting the confidence values of themultiple audio types based on the importance of the multiple audiotypes, or through weighting the effects of the multiple audio typesbased on the confidence values. Specifically, the adjusting operation1104 may be configured to consider at least one dominant audio typebased on the confidence values.

As described in Part 1, the adjusted parameter value may be smoothed.Reference may be made to Section 1.5 and Section 1.8, and detaileddescription is omitted here.

The audio type may be either content type or context type, or both. Wheninvolving the content type, the adjusting operation 1104 may beconfigured to positively correlate an equalization level with aconfidence value of short-term music, and/or negatively correlate theequalization level with a confidence value of speech. Additionally oralternatively, the adjusting operation may be configured to positivelycorrelate an equalization level with a confidence value of background,and/or negatively correlate the equalization level with a confidencevalue of noise.

When involving the context type, the adjusting operation 1104 may beconfigured to positively correlate an equalization level with aconfidence value of long-term music, and/or negatively correlate theequalization level with a confidence value of movie-like media and/orgame.

For the content type of short-term music, the adjusting operation 1104may be configured to positively correlate an equalization level with aconfidence value of short-term music without dominant sources, and/ornegatively correlate the equalization level with a confidence value ofshort-term music with dominant sources. This can be done only when theconfidence value for the short-term music is greater than a threshold.

Besides adjusting the equalization level, other aspects of an equalizermay be adjusted based on the confidence value(s) of the audio type(s) ofan audio signal. For example, the adjusting operation 1104 may beconfigured to assign an equalization level and/or equalization profileand/or spectral balance preset to each audio type.

About the specific instances of the audio types, reference may be madeto Part 1.

Similar to the embodiments of the audio processing apparatus, anycombination of the embodiments of the audio processing method and theirvariations are practical on one hand; and on the other hand, everyaspect of the embodiments of the audio processing method and theirvariations may be separate solutions. In addition, any two or moresolutions described in this section may be combined with each other, andthese combinations may be further combined with any embodiment describedor implied in the other parts of this disclosure.

Part 6: Audio Classifiers and Classifying Methods

As stated in Sections 1.1 and 1.2, the audio types discussed in thepresent application, including various hierarchical levels of contenttypes and context types, can be classified or identified with anyexisting classifying scheme, including machine-learning based methods.In this part and the subsequent part, the present application proposessome novel aspects of classifiers and methods for classifying contexttypes as mentioned in the previous parts.

6.1 Context Classifier Based on Content Type Classification

As stated in the previous parts, the audio classifier 200 is used toidentify the content type of an audio signal and/or identify the contexttype of the audio signal. Therefore, the audio classifier 200 maycomprises an audio content classifier 202 and/or an audio contextclassifier 204. When adopting existing techniques to implement the audiocontent classifier 202 and the audio context classifier 204, the twoclassifiers may be independent from each other, although they may sharesome features and thus may share some schemes for extracting thefeatures.

In this part and the subsequent Part 7, according to the novel aspectproposed in the present application, the audio context classifier 204may make use of the results of the audio content classifier 202, thatis, the audio classifier 200 may comprise: an audio content classifier202 for identifying the content type of an audio signal; and an audiocontext classifier 204 for identifying the context type of the audiosignal based on the results of the audio content classifier 202. Thus,the classification results of the audio content classifier 202 may beused by both the audio context classifier 204 and the adjusting unit 300(or the adjusting units 300A to 300D) as discussed in the previousparts. However, although not shown in the drawings, the audio classifier200 may also contain two audio content classifiers 202 to be usedrespectively by the adjusting unit 300 and the audio context classifier204.

Further, as discussed in Section 1.2, especially when classifyingmultiple audio types, either the audio content classifier 202 or theaudio context classifier 204 may be comprised a group of classifierscooperating with each other, although it is also possible to beimplemented as one single classifier.

As discussed in Section 1.1, the content type is a kind of audio typewith respect to short-term audio segments generally having a length inthe order of several to several tens of frames (such as 1 s), and thecontext type is a kind of audio type with respect to long-term audiosegments generally having a length in the order of several to severaltens of seconds (such as 10 s). Therefore, corresponding to “contenttype” and “context type”, we use “short-term” and “long-term”respectively when necessary. However, as will be discussed in thesubsequent Part 7, although the context type is for indicating theproperty of the audio signal in a relatively long timescale, it can alsobe identified based on features extracted from short-term audiosegments.

Now turn to the structures of the audio content classifier 202 and theaudio context classifier 204 with reference to FIG. 24.

As shown in FIG. 24, the audio content classifier 202 may comprise ashort-term feature extractor 2022 for extracting short-term featuresfrom short-term audio segments each comprising a sequence of audioframes; and a short-term classifier 2024 for classifying a sequence ofshort-term segments in a long-term audio segment into short-term audiotypes using respective short-term features. Both the short-term featureextractor 2022 and the short-term classifier 2024 may be implementedwith existing techniques, but also some modifications are proposed forthe short-term feature extractor 2022 in subsequent Section 6.3.

The short-term classifier 2024 may be configured to classify each of thesequence of short-term segments into at least one of the followingshort-term audio types (content types): speech, short-term music,background sound and noise, which have been explained in Section 1.1.Each of the content type may be further classified into content types onlower hierarchical level, such as discussed in Section 1.1 but notlimited thereto.

As known in the art, confidence values of the classified audio types mayalso be obtained by the short-term classifier 2024. In the presentapplication, when mentioning the operation of any classifier, it shallbe understood that confidence values are obtained at the same time ifnecessary, whether or not it is explicitly recorded. An example of audiotype classification may be found in L. Lu, H.-J. Zhang, and S. Li,“Content-based Audio Classification and Segmentation by Using SupportVector Machines”, ACM Multimedia Systems Journal 8 (6), pp. 482-492,March, 2003, which is incorporated herein in its entirety by reference.

On the other hand, as shown in FIG. 24, the audio context classifier 204may comprise a statistics extractor 2042 for calculating the statisticsof the results of the short-term classifier with respect to the sequenceof short-term segments in the long-term audio segment, as long-termfeatures; and a long-term classifier 2044 for, using the long-termfeatures, classifying the long-term audio segment into long-term audiotypes. Similarly, both the statistics extractor 2042 and the long-termclassifier 2044 may be implemented with existing techniques, but alsosome modifications are proposed for the statistics extractor 2042 insubsequent Section 6.2.

The long-term classifier 2044 may be configured to classify thelong-term audio segment into at least one of the following long-termaudio types (context types): movie-like media, long-term music, game andVoIP, which have been explained in Section 1.1. Alternatively oradditionally, The long-term classifier 2044 may be configured toclassify the long-term audio segment into VoIP or non-VoIP, which havebeen explained in Section 1.1. Alternatively or additionally, Thelong-term classifier 2044 may be configured to classify the long-termaudio segment into high-quality audio or low-quality audio, which havebeen explained in Section 1.1. In practice, various target audio typescan be chosen and trained based on the needs of application/system.

About the meaning and selection of the short-term segment and long-termsegment (as well as frame to be discussed in Section 6.3), reference maybe made to Section 1.1.

6.2 Extraction of Long-Term Features

As shown in FIG. 24, in one embodiment, only the statistics extractor2042 is used to extract long-term features from the results of theshort-term classifier 2024. As long-term features, at least one of thefollowing may be calculated by the statistics extractor 2042: mean andvariance of confidence values of the short-term audio types of theshort-term segments in the long-term segment to be classified, the meanand the variance weighted by the importance degrees of the short-termsegments, occurrence frequency of each short-term audio type andfrequency of transitions between different short-term audio types in thelong-term segment to be classified.

We illustrate in FIG. 25 the mean of the speech and short-term musicconfidence values in each short-term segment (of a length of 1s). Forcomparison, the segments are extracted from three different audiocontexts: movie-like media (FIG. 25(A)), long-term music (FIG. 25(B)),and VoIP (FIG. 25(C)). It can be observed that for movie-like mediacontext, high confidence values are gained either for speech type or formusic type and it alternates between these two audio types frequently.By contrast, the segment of long-term music gives a stable and highshort-term music confidence value and a relatively stable and low speechconfidence value. Whereas the segment of VoIP gives a stable and lowshort-term music confidence value, but gives fluctuating speechconfidence values because of the pauses during the VoIP conversation.

The variance of the confidence values for each audio type is also animportant feature for classifying different audio contexts. FIG. 26gives histograms of the variance of the confidence values of speech,short-term music, background and noise in movie-like media, long-termmusic and VoIP audio contexts (the abscissa is the variance ofconfidence values in a dataset, and the ordinate is the number ofoccurrences of each bin of variance values s in the dataset, which canbe normalized to indicate the occurrence probability of each bin ofvariance values). For movie-like media, all the variances of confidencevalue of speech, short-term music and background are relatively high andwidely distributed, indicating that the confidence values of those audiotypes are changing intensively; For long-term music, all the variancesof confidence value of speech, short-term music, background and noiseare relatively low and narrowly distributed, indicating that theconfidence values of those audio types are keeping stable: speechconfidence value keeps constantly low and music confidence value keepsconstantly high; For VoIP, the variances of confidence value ofshort-term music are low and narrowly distributed, whereas that ofspeech are relatively widely distributed, which is due to frequentpauses during VoIP conversations.

About the weights used in calculating the weighted mean and variance,they are determined based on each short-term segment's importancedegree. The important degree of a short-term segment may be measured byits energy or loudness, which can be estimated with many existingtechniques.

The occurrence frequency of each short-term audio type in the long-termsegment to be classified is the count of each audio type to which theshort-term segments in the long-term segment have been classified,normalized with the length of the long-term segment.

The frequency of transitions between different short-term audio types inthe long-term segment to be classified is the count of audio typechanges between adjacent short-term segments in the long-term segment tobe classified, normalized with the length of the long-term segment.

When discussing the mean and the variance of the confidence values withreference to FIG. 25, the occurrence frequency of each short-term audiotype and the transition frequency among those different short-term audiotypes are also touched in fact. These features are also highly relevantto audio context classification. For example, the long-term music mostlycontains short-term music audio type so it has high occurrence frequencyof short-term music, whereas the VoIP mostly contains speech and pausesso it has high occurrence frequency of speech or noise. For anotherexample, the movie-like media transits among different short-term audiotypes more frequently than long-term music or VoIP does, so it generallyhas a higher transition frequency among short-term music, speech andbackground; VoIP usually transits between speech and noise morefrequently than the others do, so it generally has a higher transitionfrequency between speech and noise.

Generally, we assume the long-term segments are of the same length inthe same application/system. If this is the case, then the occurrencecount of each short-term audio type, and the transition count betweendifferent short-term audio types in the long-term segment may bedirectly used without normalization. If the length of the long-termsegment is variable, then the occurrence frequency and the frequency oftransitions as mentioned above shall be used. And the claims in thepresent application shall be construed as covering both situations.

Additionally or alternatively, the audio classifier 200 (or the audiocontext classifier 204) may further comprise a long-term featureextractor 2046 (FIG. 27) for extracting further long-term features fromthe long-term audio segment based on the short-term features of thesequence of short-term segments in the long-term audio segment. In otherwords, the long-term feature extractor 2046 does not use theclassification results of the short-term classifier 2024, but directlyuse the short-term features extracted by the short-term featureextractor 2022 to derive some long-term features to be used by thelong-term classifier 2044. The long-term feature extractor 2046 and thestatistics extractor 2042 may be used independently or jointly. In otherwords, the audio classifier 200 may comprise either the long-termfeature extractor 2046 or the statistics extractor 2042, or both.

Any features can be extracted by the long-term feature extractor 2046.In the present application, it is proposed to calculate, as thelong-term features, at least one of the following statistics of theshort-term features from the short-term feature extractor 2022: mean,variance, weighted mean, weighted variance, high average, low average,and ratio (contrast) between the high average and the low average.

Mean and variance of the short-term features extracted from theshort-term segments in the long-term segment to be classified;

Weighted mean and variance of the short-term features extracted from theshort-term segments in the long-term segment to be classified. Theshort-term features are weighted based on each short-term segment'simportance degree that is measured with its energy or loudness asmentioned just now;

High average: an average of selected short-term features extracted fromthe short-term segments in the long-term segment to be classified. Theshort-term features are selected when meeting at least one of thefollowing conditions: greater than a threshold; or within apredetermined proportion of short-term features not lower than all theother short-term features, for example, the highest 10% of theshort-term features;

Low average: an average of selected short-term features extracted fromthe short-term segments in the long-term segment to be classified. Theshort-term features are selected when satisfying at least one of thefollowing conditions: smaller than a threshold; or within apredetermined proportion of the short-term features not higher than allthe other short-term features, for example, the lowest 10% of theshort-term features; and

Contrast: a ratio between the high average and the low average torepresent the dynamic of the short-term features in a long-term segment.

The short-term feature extractor 2022 may be implemented with existingtechniques, and any features can be extracted thereby. Nevertheless,some modifications are proposed for the short-term feature extractor2022 in subsequent Section 6.3.

6.3 Extraction of Short-Term Features

As shown in FIG. 24 and FIG. 27, the short-term feature extractor 2022may be configured to extract, as short-term features, at least one ofthe following features directly from each short-term audio segment:rhythmic characteristics, interruptions/mutes characteristics andshort-term audio quality features.

The rhythmic characteristics may include rhythm strength, rhythmregularity, rhythm clarity (see L. Lu, D. Liu, and H.-J. Zhang.“Automatic mood detection and tracking of music audio signals”. IEEETransactions on Audio, Speech, and Language Processing, 14(1):5-18,2006, which is incorporated herein in its entirety by reference) and 2Dsub-band modulation (M. F McKinney and J. Breebaart. “Features for audioand music classification”, Proc. ISMIR, 2003, which is incorporatedherein in its entirety by reference).

The interruptions/mutes characteristics may include speechinterruptions, sharp declines, mute length, unnatural silence, mean ofunnatural silence, total energy of unnatural silence, etc.

The short-term audio quality features are audio quality features withrespect to short term segments, which are similar to audio qualityfeatures extracted from audio frames, which are to be discussed below.

Alternatively or additionally, as shown in FIG. 28, the audio classifier200 may comprise a frame-level feature extractor 2012 for extractingframe-level features from each of the sequence of audio frames comprisedin a short-term segment, and the short-term feature extractor 2022 maybe configured to calculate short-term features based on the frame-levelfeatures extracted from the sequence of audio frames.

As pre-processing, the input audio signal may be down-mixed to a monoaudio signal. The pre-processing is unnecessary if the audio signal isalready a mono signal. It is then divided into frames with a predefinedlength (typically 10 to 25 milliseconds). Correspondingly, frame-levelfeatures are extracted from each frame.

The frame-level feature extractor 2012 may be configured to extract atleast one of the following features: features characterizing propertiesof various short-term audio types, cutoff frequency, staticsignal-noise-ratio (SNR) characteristics, segmental signal-noise-ratio(SNR) characteristics, basic speech descriptors, and vocal tractcharacteristics.

The features characterizing properties of various short-term audio types(especially speech, short-term music, background sound and noise) maycomprise at least one of the following features: frame energy, sub-bandspectral distribution, spectral flux, Mel-frequency Cepstral Coefficient(MFCC), bass, residual information, Chroma feature and zero-crossingrate.

For detail of MFCC, reference may be made to L. Lu, H.-J. Zhang, and S.Li, “Content-based Audio Classification and Segmentation by UsingSupport Vector Machines”, ACM Multimedia Systems Journal 8 (6), pp.482-492, March, 2003, which is incorporated herein in its entirety byreference. For detail of Chroma feature, reference may be made to G. H.Wakefield, “Mathematical representation of joint time Chromadistributions” in SPIE, 1999, which is incorporated herein in itsentirety by reference.

The cutoff frequency represents an audio signal's highest frequencyabove which the energy of the content is close to zero. It is designedto detect band limited content, which is useful in this application foraudio context classification. A cutoff frequency is usually caused bycoding, as most coders discard high frequencies at low or mediumbitrates. For example, MP3 codec has a cutoff frequency of 16 kHz at 128kbps; For another example, many popular VoIP codecs have a cutofffrequency of 8 kHz or 16 kHz.

Besides the cutoff frequency, signal degradation during the audioencoding process is considered as another characteristic fordifferentiating various audio contexts such as VoIP vs. non-VoIPcontexts, high-quality vs. low-quality audio contexts. The featuresrepresenting the audio quality, such as those for objective speechquality assessment (see Ludovic Malfait, Jens Berger, and MartinKastner, “P. 563—The ITU-T Standard for Single-Ended Speech QualityAssessment”, IEEE Transaction on Audio, Speech, and Language Processing,VOL. 14, NO. 6, November 2006, which in incorporated herein in itsentirety by reference), may be further extracted in multiple levels tocapture richer characteristics. Examples of the audio quality featuresinclude:

-   -   a) Static SNR characteristics including estimated background        noise level, spectral clarity, etc.    -   b) Segmental SNR characteristics including spectral level        deviation, spectral level range, relative noise floor, etc.    -   c) Basic speech descriptors including pitch average, speech        section level variation, speech level, etc.    -   d) Vocal tract characteristics including robotization, pitch        cross power, etc.

For deriving the short-term features from the frame-level features, theshort-term feature extractor 2022 may be configured to calculatestatistics of the frame-level features, as the short-term features.

Examples of the statistics of the frame-level features include mean andstandard deviation, which captures the rhythmic properties todifferentiate various audio types, such as short-term music, speech,background and noise. For example, speech usually alternates betweenvoiced and unvoiced sounds at a syllable rate whereas music does not,indicating that the variation of the frame-level features of speech isusually larger than that of music.

Another example of the statistics is the weighted average of theframe-level features. For example, for the cutoff frequency, theweighted average among the cutoff frequencies derived from every audioframes in a short-term segment, with the energy or loudness of eachframe as weight, would be the cutoff frequency for that short-termsegment.

Alternatively or additionally, as shown in FIG. 29, the audio classifier200 may comprise a frame-level feature extractor 2012 for extractingframe-level features from audio frames and a frame-level classifier 2014for classifying each of the sequence of audio frames into frame-levelaudio types using respective frame-level features, wherein theshort-term feature extractor 2022 may be configured to calculate theshort-term features based on the results of the frame-level classifier2014 with respect to the sequence of audio frames.

In other words, in addition to the audio content classifier 202 and theaudio context classifier 204, the audio classifier 200 may furthercomprise a frame classifier 201. In such an architecture, the audiocontent classifier 202 classifies a short-term segment based on theframe-level classification results of the frame classifier 201, and theaudio context classifier 204 classifies a long-term segment based on theshort-term classification results of the audio content classifier 202.

The frame-level classifier 2014 may be configured to classify each ofthe sequence of audio frames into any classes, which may be referred toas “frame-level audio types”. In one embodiment, the frame-level audiotypes may have an architecture similar to the architecture of thecontent types discussed hereinbefore and have also meaning similar tothe content types, and the only difference is the frame-level audiotypes and the content types are classified at different levels of theaudio signal, that is frame-level and short-term segment level. Forexample, the frame-level classifier 2014 may be configured to classifyeach of the sequence of audio frames into at least one of the followingframe-level audio types: speech, music, background sound and noise. Onthe other hand, the frame-level audio types may also have anarchitecture partly or completely different from the architecture of thecontent types, more suitable to the frame-level classification, and moresuitable to be used as the short-term features for short-termclassification. For example, the frame-level classifier 2014 may beconfigured to classify each of the sequence of audio frames into atleast one of the following frame-level audio types: voiced, unvoiced,and pause.

About how to derive short-term features from the results of theframe-level classification, a similar scheme may be adopted by referringto the description in Section 6.2.

As an alternative, both short-term features based on the results of theframe-level classifier 2014 and short-term features directly based onthe frame-level features obtained by the frame-level feature extractor2012 may be used by the short-term classifier 2024. Therefore, theshort-term feature extractor 2022 may be configured to calculate theshort-term features based on both the frame-level features extractedfrom the sequence of audio frames and the results of the frame-levelclassifier with respect to the sequence of audio frames.

In other words, the frame-level feature extractor 2012 may be configuredto calculate both statistics similar to those discussed in Section 6.2and those short-term features described in connection with FIG. 28,including at least one of the following features: featurescharacterizing properties of various short-term audio types, cutofffrequency, static signal-noise-ratio characteristics, segmentalsignal-noise-ratio characteristics, basic speech descriptors, and vocaltract characteristics.

For working in real time, in all the embodiments the short-term featureextractor 2022 may be configured to work on the short-term audiosegments formed with a moving window sliding in the temporal dimensionof the long-term audio segment at a predetermined step length. About themoving window for the short-term audio segment, as well as the audioframe and the moving window for the long-term audio segment, referencemay be made to Section 1.1 for detail.

6.4 Combination of Embodiments and Application Scenarios

Similar to Part 1, all the embodiments and variants there of discussedabove may be implemented in any combination thereof, and any componentsmentioned in different parts/embodiments but having the same or similarfunctions may be implemented as the same or separate components.

For example, any two or more of the solutions described in Sections 6.1to 6.3 may be combined with each other. And any of the combinations maybe further combined with any embodiment described or implied in Parts1-5 and the other parts that will be described later. Especially, thetype smoothing unit 712 discussed in Part 1 may be used in this part asa component of the audio classifier 200, for smoothing the results ofthe frame classifier 2014, or the audio content classifier 202, or theaudio context classifier 204. Further, the timer 916 may also serve as acomponent of the audio classifier 200 to avoid abrupt change of theoutput of the audio classifier 200.

6.5 Audio Classifying Method

Similar to Part 1, in the process of describing the audio classifier inthe embodiments hereinbefore, apparently disclosed are also someprocesses or methods. Hereinafter a summary of these methods is givenwithout repeating some of the details already discussed hereinbefore.

In one embodiment, as shown in FIG. 30, an audio classifying method isprovided. To identify the long-term audio type (that is context type) ofa long-term audio segment comprised of a sequence of short-term audiosegments (either overlapped or non-overlapped with each other), theshort-term audio segments are firstly classified (operation 3004) intoshort-term audio types, that is content types, and long-term featuresare obtained by calculating (operation 3006) the statistics of theresults of classifying operation with respect to the sequence ofshort-term segments in the long-term audio segment. Then the long-termclassifying (operation 3008) may be performed using the long-termfeatures. The short-term audio segment may comprise a sequence of audioframes. Of course, for identifying the short-term audio type of theshort-term segments, short-term features need be extracted from them(operation 3002).

The short-term audio types (content types) may include but is notlimited to speech, short-term music, background sound and noise.

The long-term features may include but is not limited to: mean andvariance of confidence values of the short-term audio types, the meanand the variance weighted by the importance degrees of the short-termsegments, occurrence frequency of each short-term audio type andfrequency of transitions between different short-term audio types.

In a variant, as shown in FIG. 31, further long-term features may beobtained (operation 3107) directly based on the short-term features ofthe sequence of short-term segments in the long-term audio segment. Suchfurther long-term features may include but are not limited to thefollowing statistics of the short-term features: mean, variance,weighted mean, weighted variance, high average, low average, and ratiobetween the high average and the low average.

There are different ways for extracting the short-term features. One isto directly extract the short-term features from the short-term audiosegment to be classified. Such features include but are not limited torhythmic characteristics, interruptions/mutes characteristics andshort-term audio quality features.

The second way is to extract frame-level features from the audio framescomprised in each short-term segment (operation 3201 in FIG. 32), andthen calculate short-term features based on the frame-level features,such as calculate statistics of the frame-level features as theshort-term features. The frame-level features may comprise but are notlimited to: features characterizing properties of various short-termaudio types, cutoff frequency, static signal-noise-ratiocharacteristics, segmental signal-noise-ratio characteristics, basicspeech descriptors, and vocal tract characteristics. The featurescharacterizing properties of various short-term audio types may furthercomprise frame energy, sub-band spectral distribution, spectral flux,Mel-frequency Cepstral Coefficient, bass, residual information, Chromafeature and zero-crossing rate.

The third way is to extract the short-term features in a manner similarto extraction of the long-term features: after extracting theframe-level features from audio frames in a short-term segment to beclassified (operation 3201), classifying each audio frame intoframe-level audio types using respective frame-level features (operation32011 in FIG. 33); and the short-term features may be extracted(operation 3002) by calculating the short-term features based on theframe-level audio types (optionally including the confidence values).The frame-level audio types may have properties and an architecturesimilar to the short-term audio type (content type), and may alsoinclude speech, music, background sound and noise.

The second way and the third way may be combined together as shown bythe dashed arrow in FIG. 33.

As discussed in Part 1, both short-term audio segments and long-termaudio segments may be sampled with moving windows. That is, theoperation of extracting short-term features (operation 3002) may beperformed on short-term audio segments formed with a moving windowsliding in the temporal dimension of the long-term audio segment at apredetermined step length, and the operation of extracting long-termfeatures (operation 3107) and the operation of calculating statistics ofshort-term audio types (operation 3006) may also be performed onlong-term audio segments formed with a moving window sliding in thetemporal dimension of the audio signal at a predetermined step length.

Similar to the embodiments of the audio processing apparatus, anycombination of the embodiments of the audio processing method and theirvariations are practical on one hand; and on the other hand, everyaspect of the embodiments of the audio processing method and theirvariations may be separate solutions. In addition, any two or moresolutions described in this section may be combined with each other, andthese combinations may be further combined with any embodiment describedor implied in the other parts of this disclosure. Especially, as alreadydiscussed in Section 6.4, the smoothing schemes and the transitionscheme of audio types may be a part of the audio classifying methoddiscussed here.

Part 7: VoIP Classifiers and Classifying Methods

In Part 6 a novel audio classifier is proposed for classifying an audiosignal into audio context types at least partly based on the results ofcontent type classification. In the embodiments discussed in Part 6,long-term features are extracted from a long-term segment of a length ofseveral to several tens of seconds, thus the audio contextclassification may cause long latency. It is desired that the audiocontext may also be classified in real time or nearly in real time, suchas at the short-term segment level.

7.1 Context Classification Based on Short-Term Segment

Therefore, as shown in FIG. 34, an audio classifier 200A is provided,comprising audio content classifier 202A for identifying a content typeof a short-term segment of an audio signal, and an audio contextclassifier 204A for identifying a context type of the short-term segmentat least partly based on the content type identified by the audiocontent classifier.

Here the audio content classifier 202A may adopt the techniques alreadymentioned in Part 6, but may also adopt different techniques as will bediscussed below in Section 7.2. Also, the audio context classifier 204Amay adopt the techniques already mentioned in Part 6, with a differencethat the context classifier 204A may directly use the results of theaudio content classifier 202A, rather than use the statistics of theresults from the audio content classifier 202A since both the audiocontext classifier 204A and the audio content classifier 202A areclassifying the same short-term segment. Further, similar to Part 6, inaddition to the results from the audio content classifier 202A, theaudio context classifier 204A may use other features directly extractedfrom the short-term segment. That is, the audio context classifier 204Amay be configured to classify the short-term segment based on amachine-learning model by using, as features, the confidence values ofthe content types of the short-term segment and other features extractedfrom the short-term segment. About the features extracted from theshort-term segment, reference may be made to Part 6.

The audio content classifier 200A may simultaneously label theshort-term segment as more audio types than VoIP speech/noise and/ornon-VoIP speech/noise (VoIP speech/noise and non-VoIP speech/noise willbe discussed below in Section 7.2), and each of the multiple audio typesmay have its own confidence value as discussed in Section 1.2. This canachieve better classification accuracy since richer information can becaptured. For example, joint information of the confidence values ofspeech and short-term music reveals to what extent the audio content islikely to be a mixture of speech and background music so that it can bediscriminated from pure VoIP content.

7.2 Classification Using VoIP Speech and VoIP Noise

This aspect of the present application is especially useful in aVoIP/non-VoIP classification system, which would be required to classifythe current short-term segment for short decision latency.

For this purpose, as shown in FIG. 34, the audio classifier 200A isspecially designed for VoIP/non-VoIP classification. For classifyingVoIP/non-VoIP, a VoIP speech classifier 2026 and/or a VoIP noiseclassifier are developed to generate intermediate results for finalrobust VoIP/non-VoIP classification by the audio context classifier204A.

A VoIP short-term segment would contain VoIP speech and VoIP noisealternatively. It is observed that high accuracy can be achieved toclassify a short-term segment of speech into VoIP speech or non-VoIPspeech, but not so for classifying a short-term segment of noise intoVoIP noise or non-VoIP noise. Thus, it can be concluded that it willblur the discriminability by directly classifying the short-term segmentinto VoIP (comprising VoIP speech and VoIP noise but with VoIP speechand VoIP noise not specifically identified) and non-VoIP withoutconsidering the difference between speech and noise and thus with thefeatures of these two content types (speech and noise) mixed together.

It is reasonable for classifiers to achieve higher accuracies for VoIPspeech/non-VoIP speech classification than for VoIP noise/non-VoIP noiseclassification as speech contains more information than noise does andsuch features as cutoff frequency are more effective for classifyingspeech. According to the weight ranking obtained from adaBoost trainingprocess, the top weighted short-term features for VoIP/non-VoIP speechclassification are: standard deviation of logarithm energy, cutofffrequency, standard deviation of rhythmic strength, and standarddeviation of spectral flux. The standard deviation of logarithm energy,standard deviation of rhythmic strength, and standard deviation ofspectral flux are generally higher for VoIP speech than for non-VoIPspeech. One probable reason is that many short-term speech segments in anon-VoIP context such as a movie-like media or a game are usually mixedwith other sounds such as background music or sound effect, of which thevalues of the above features are lower. Meanwhile, the cutoff feature isgenerally lower for VoIP speech than for non-VoIP speech, whichindicates the low cutoff frequency introduced by the many popular VoIPcodecs.

Therefore, in one embodiment, the audio content classifier 202A maycomprise a VoIP speech classifier 2026 for classifying the short-termsegment into the content type VoIP speech or the content type non-VoIPspeech; and the audio context classifier 204A may be configured toclassify the short-term segment into the context type VoIP or thecontext type non-VoIP based on confidence values of VoIP speech andnon-VoIP speech.

In another embodiment, the audio content classifier 202A may furthercomprise a VoIP noise classifier 2028 for classifying the short-termsegment into the content type VoIP noise or the content type non-VoIPnoise; and the audio context classifier 204A may be configured toclassify the short-term segment into the context type VoIP or thecontext type non-VoIP based on confidence values of VoIP speech,non-VoIP speech, VoIP noise and non-VoIP noise.

The content types of VoIP speech, non-VoIP speech, VoIP noise andnon-VoIP noise may be identified with existing techniques as discussedin Part 6, Section 1.2 and Section 7.1.

Alternatively, the audio content classifier 202A may have a hierarchicalstructure as shown in FIG. 35. That is, we take advantage of the resultsfrom a speech/noise classifier 2025 to first classify the short-termsegment into speech or noise/background.

On the basis of the embodiment using merely VoIP speech classifier 2026,if a short-term segment is determined as speech by the speech/noiseclassifier 2025 (in such a situation it is just a speech classifier),then the VoIP speech classifier 2026 continues to classify whether it isVoIP speech or non-VoIP speech, and calculates the binary classificationresult; Otherwise it may be regarded that the confidence value of VoIPspeech is low, or the decision on the VoIP speech is uncertain.

On the basis of the embodiment using merely VoIP noise classifier 2028,if the short-term segment is determined as noise, by the speech/noiseclassifier 2025 (in such a situation it is just a noise (background)classifier), then the VoIP noise classifier 2028 continues to classifyit into VoIP noise or non-VoIP noise, and calculate the binaryclassification result. Otherwise it may be regarded that the confidencevalue of VoIP noise is low, or the decision on the VoIP noise isuncertain.

Here, since generally speech is an informative content type andnoise/background is an interfering content type, even if a short-termsegment is not a noise, in the embodiment in the previous paragraph wecan not determine definitely that the short-term segment is not of thecontext type VoIP. While if a short-term segment is not a speech, in theembodiment merely using the VoIP speech classifier 2026 it is probablynot the context type VoIP. Therefore, generally the embodiment usingmerely VoIP speech classifier 2026 may be realized independently, whilethe other embodiment using merely VoIP noise classifier 2028 may be usedas a supplementary embodiment cooperating with, for example, theembodiment using the VoIP speech classifier 2026.

That is, both VoIP speech classifier 2026 and VoIP noise classifier 2028may be used. If a short-term segment is determined as speech by thespeech/noise classifier 2025, then the VoIP speech classifier 2026continues to classify whether it is VoIP speech or non-VoIP speech, andcalculates the binary classification result. If the short-term segmentis determined as noise by the speech/noise classifier 2025, then theVoIP noise classifier 2028 continues to classify it into VoIP noise ornon-VoIP noise, and calculate the binary classification result.Otherwise, it may be regarded that short-term segment may be classifiedas non-VoIP.

The implementation of the speech/noise classifier 2025, the VoIP speechclassifier 2026 and the VoIP noise classifier 2028 may adopt anyexisting techniques, and may be the audio content classifier 202discussed in Parts 1-6.

If the audio content classifier 202A implemented according to abovedescription finally classifies a short-term segment into none of speech,noise and background, or none of VoIP speech, non-VoIP speech, VoIPnoise and non-VoIP noise, meaning all the relevant confidence values arelow, then the audio content classifier 202A (and the audio contextclassifier 204A) may classify the short-term segment as non-VoIP.

For classifying the short-term segment into the context types of VoIP ornon-VoIP based on the results of the VoIP speech classifier 2026 and theVoIP noise classifier 2028, the audio context classifier 204A may adoptmachine-learning based techniques as discussed in Section 7.1, and as amodification, more features may be used, including short-term featuresdirectly extracted from the short-term segment and/or results of otheraudio content classifier(s) directed to other content types than VoIPrelated content types, as already discussed in Section 7.1.

Besides the above described machine-learning based techniques, analternative approach to VoIP/non-VoIP classification can be a heuristicrule taking advantage of domain knowledge and utilizing theclassification results in connection with VoIP speech and VoIP noise. Anexemplary of such heuristic rules will be illustrated below.

If the current short-term segment of time t is determined as VoIP speechor non-VoIP speech, the classification result is directly taken as theVoIP/non-VoIP classification result since VoIP/non-VoIP speechclassification is robust as discussed before. That is, if the short-termsegment is determined as VoIP speech, then it is the context type VoIP;if the short-term segment is determined as non-VoIP speech, then it isthe context type non-VoIP.

When the VoIP speech classifier 2026 makes a binary decision regardingVoIP speech/non-VoIP speech with respect to speech determined by thespeech/noise classifier 2025 as mentioned above, the confidence valuesof VoIP speech and non-VoIP speech might be complementary, that is, thesum thereof is 1 (if 0 represents 100% not and 1 represents 100% yes),and the thresholds of confidence value for differentiating VoIP speechand non-VoIP speech may indicate actually the same point. If the VoIPspeech classifier 2026 is not a binary classifier, the confidence valuesof VoIP speech and non-VoIP speech might be not complementary, and thethresholds of confidence value for differentiating VoIP speech andnon-VoIP speech may not necessarily indicate the same point.

However, in the case where the VoIP speech or non-VoIP speech confidenceis close to and fluctuates around the threshold, the VoIP/non-VoIPclassification results are possible to switch too frequently. To avoidsuch fluctuation, a buffer scheme may be provided: both thresholds forVoIP speech and non-VoIP speech may be set larger, so that it is not soeasy to switch from the present content type to the other content type.For ease of description, we may convert the confidence value fornon-VoIP speech to the confidence value of VoIP speech. That is, if theconfidence value is high, the short-term segment is regarded as closerto VoIP speech, and if the confidence value is low, the short-termsegment is regarded as closer to non-VoIP speech. Although fornon-binary classifier as described above a high confidence value ofnon-VoIP speech does not necessarily mean a low confidence value of VoIPspeech, such simplification may well reflect the essence of the solutionand the relevant claims described with language of binary classifiersshall be construed as covering equivalent solutions for non-binaryclassifiers.

The buffer scheme is shown in FIG. 36. There is a buffer area betweentwo thresholds Th1 and Th2 (Th1>=Th2). When the confidence value v(t) ofVoIP speech falls in the area, the context classification will notchange, as shown by the arrows on the left and right sides in FIG. 36.Only when the confidence value v(t) is greater than the larger thresholdTh1, will the short-term segment be classified as VoIP (as shown by thearrow on the bottom in FIG. 36); and only when the confidence value isnot greater than the smaller threshold Th2, will the short-term segmentbe classified as non-VoIP (as shown by the arrow on the top in FIG. 36).

If the VoIP noise classifier 2028 is used instead, the situation issimilar. For making the solution more robust, the VoIP speech classifier2026 and the VoIP noise classifier 2028 may be used jointly. Then, theaudio context classifier 204A may be configured to: classify theshort-term segment as the context type VoIP if the confidence value ofVoIP speech is greater than a first threshold or if the confidence valueof VoIP noise is greater than a third threshold; classify the short-termsegment as the context type non-VoIP if the confidence value of VoIPspeech is not greater than a second threshold, wherein the secondthreshold not larger than the first threshold, or if the confidencevalue of VoIP noise is not greater than a fourth threshold, wherein thefourth threshold not larger than the third threshold; otherwise classifythe short-term segment as the context type for the last short-termsegment.

Here, the first threshold may be equal to the second threshold, and thethird threshold may be equal to the fourth threshold, especially for butnot limited to binary VoIP speech classifier and binary VoIP noiseclassifier. However, since generally the VoIP noise classificationresult is not so robust, it would be better if the third and the fourththresholds are not equal to each other, and both should be far from 0.5(0 indicates high confidence to be non-VoIP noise and 1 indicates highconfidence to be VoIP noise).

7.3 Smoothing Fluctuation

For avoiding rapid fluctuation, another solution is to smooth theconfidence value as determined by the audio content classifier.Therefore, as shown in FIG. 37, a type smoothing unit 203A may becomprised in the audio classifier 200A. For the confidence value of eachof 4 VoIP related content types as discussed before, the smoothingschemes discussed in Section 1.3 may be adopted.

Alternatively, similar to Section 7.2, VoIP speech and non-VoIP speechmay regarded as a pair having complementary confidence values; and VoIPnoise and non-VoIP noise may also be regarded a pair havingcomplementary confidence values. In such a situation, only one out ofeach pair needs to be smoothed, and the smoothing schemes discussed inSection 1.3 may be adopted.

Take the confidence value of VoIP speech as an example, the formula (3)may be rewritten as:v(t)=β·v(t−1)+(1−β)·voipSpeechConf(t)  (3″)where v(t) is the smoothed VoIP speech confidence value at time t,v(t−1) is the smoothed VoIP speech confidence value at the last time,and voipSpeechConf is the VoIP speech confidence at current time tbefore smoothing, α is a weighting coefficient.

In a variant, if there is a speech/noise classifier 2025 as describedabove, if the confidence value of speech for a short-segment is low,then the short-term segment cannot be classified as VoIP speechrobustly, and we can directly set voipSpeechConf (t)=v(t−1) withoutmaking the VoIP speech classifier 2026 actually work.

Alternatively, in the situation described above, we could setvoipSpeechConf (t)=0.5 (or other value not higher than 0.5, such as0.4-0.5) indicating an uncertain case (here, confidence=1 indicates ahigh confidence that it is VoIP and confidence=0 indicates a highconfidence that it is not a VoIP).

Therefore, according to the variant, as shown in FIG. 37, the audiocontent classifier 200A may further comprise a speech/noise classifier2025 for identifying content type of speech of the short-term segment,and the type smoothing unit 203A may be configured to set the confidencevalue of VoIP speech for the present short-term segment before smoothingas a predetermined confidence value (such as 0.5 or other value, such as0.4-0.5) or the smoothed confidence value of the last short-term segmentwhere the confidence value for the content type speech as classified bythe speech/noise classifier is lower than a fifth threshold. In such asituation, the VoIP speech classifier 2026 may or may not work.Alternatively the setting of the confidence value may be done by theVoIP speech classifier 2026, this is equivalent to the solution wherethe work is done by the type smoothing unit 203A, and the claim shall beconstrued as covering both situations. In addition, here we use thelanguage “the confidence value for the content type speech as classifiedby the speech/noise classifier is lower than a fifth threshold”, but thescope of protection is not limited thereto, and it is equivalent to thesituation where the short-term segment is classified into other contenttypes than speech.

For the confidence value of VoIP noise, the situation is similar anddetailed description is omitted here.

For avoiding rapid fluctuation, yet another solution is to smooth theconfidence value as determined by the audio context classifier 204A, andthe smoothing schemes discussed in Section 1.3 may be adopted.

For avoiding rapid fluctuation, still another solution is to delay thetransition of the context type between VoIP and non-VoIP, and the samescheme as that described in Section 1.6 may be used. As described inSection 1.6, the timer 916 may be outside the audio classifier or withinthe audio classifier as a part thereof. Therefore, as shown in FIG. 38,the audio classifier 200A may further comprise the timer 916. And theaudio classifier is configured to continue to output the present contexttype until the length of the lasting time of a new context type reachesa sixth threshold (context type is an instance of audio type). Byreferring to Section 1.6, detailed description may be omitted here.

In addition or alternatively, as another scheme for delaying thetransition between VoIP and non-VoIP, the first and/or second thresholdas described before for VoIP/non-VoIP classification may be differentdepending on the context type of the last short-term segment. That is,the first and/or second threshold becomes larger when the context typeof the new short-term segment is different from the context type of thelast short-term segment, while becomes smaller when the context type ofthe new short-term segment is the same as the context type of the lastshort-term segment. By this way, the context type tends to be maintainedat the current context type and thus abrupt fluctuation of the contexttype may be suppressed to some extent.

7.4 Combination of Embodiments and Application Scenarios

Similar to Part 1, all the embodiments and variants there of discussedabove may be implemented in any combination thereof, and any componentsmentioned in different parts/embodiments but having the same or similarfunctions may be implemented as the same or separate components.

For example, any two or more of the solutions described in Sections 7.1to 7.3 may be combined with each other. And any of the combinations maybe further combined with any embodiment described or implied in Parts1-6. Especially, the embodiments discussed in this part and anycombination thereof may be combined with the embodiments of the audioprocessing apparatus/method or the volume leveler controller/controllingmethod discussed in Part 4.

7.5 VoIP Classifying Method

Similar to Part 1, in the process of describing the audio classifier inthe embodiments hereinbefore, apparently disclosed are also someprocesses or methods. Hereinafter a summary of these methods is givenwithout repeating some of the details already discussed hereinbefore.

In one embodiment as shown FIG. 39, an audio classifying method includesidentifying a content type of a short-term segment of an audio signal(operation 4004), then identifying a context type of the short-termsegment at least partly based on the content type as identified(operation 4008).

For identifying the context type of an audio signal dynamically andfast, the audio classifying method in this part is especially useful inidentifying the context type VoIP and non-VoIP. In such a situation, theshort-term segment may be firstly classified into the content type VoIPspeech or the content type non-VoIP speech, and the operation ofidentifying the context type is configured to classify the short-termsegment into the context type VoIP or the context type non-VoIP based onconfidence values of VoIP speech and non-VoIP speech.

Alternatively, the short-term segment may be firstly classified into thecontent type VoIP noise or the content type non-VoIP noise, and theoperation of identifying the context type may be configured to classifythe short-term segment into the context type VoIP or the context typenon-VoIP based on confidence values of VoIP noise and non-VoIP noise.

The speech and the noise may be considered jointly. In such a situation,the operation of identifying the context type may be configured toclassify the short-term segment into the context type VoIP or thecontext type non-VoIP based on confidence values of VoIP speech,non-VoIP speech, VoIP noise and non-VoIP noise.

For identifying the context type of the short-term segment, amachine-learning model may be used, taking both the confidence values ofthe content types of the short-term segment and other features extractedfrom the short-term segment as features.

The operation of identifying the context type may also be realized basedon heuristic rules. When only VoIP speech and non-VoIP speech areinvolved, the heuristic rule is like this: classify the short-termsegment as the context type VoIP if the confidence value of VoIP speechis greater than a first threshold; classify the short-term segment asthe context type non-VoIP if the confidence value of VoIP speech is notlarger than a second threshold, wherein the second threshold not largerthan the first threshold; otherwise, classify the short-term segment asthe context type for the last short-term segment.

The heuristic rule for the situation where only VoIP noise and non-VoIPnoise are involved is similar.

When both speech and noise are involved, the heuristic rule is likethis: classify the short-term segment as the context type VoIP if theconfidence value of VoIP speech is greater than a first threshold or ifthe confidence value of VoIP noise is greater than a third threshold;classify the short-term segment as the context type non-VoIP if theconfidence value of VoIP speech is not greater than a second threshold,wherein the second threshold not larger than the first threshold, or ifthe confidence value of VoIP noise is not greater than a fourththreshold, wherein the fourth threshold not larger than the thirdthreshold; otherwise, classify the short-term segment as the contexttype for the last short-term segment.

The smoothing scheme discussed in Section 1.3 and Section 1.8 may beadopted here and detailed description is omitted. As a modification tothe smoothing scheme described in

Section 1.3, before the smoothing operation 4106, the method may furthercomprise identifying the content type speech from the short-term segment(operation 40040 in FIG. 40), wherein the confidence value of VoIPspeech for the present short-term segment before smoothing is set as apredetermined confidence value or the smoothed confidence value of thelast short-term segment (operation 40044 in FIG. 40) where theconfidence value for the content type speech is lower than a fifththreshold (“N” in operation 40041).

If otherwise the operation of identifying the content type speechrobustly judges the short-term segment as speech (“Y” in operation40041), then the short-term segment is further classified into VoIPspeech or non-VoIP speech (operation 40042), before the smoothingoperation 4106.

In fact, even without using the smoothing scheme, the method may alsoidentify the content type speech and/or noise first, when the short-termsegment is classified as speech or noise, further classification isimplemented to classify the short-term segment into one of VoIP speechand non-VoIP speech, or one of VoIP noise and non-VoIP noise. Then theoperation of identifying the context type is made.

As mentioned in Section 1.6 and Section 1.8, the transition schemediscussed therein may be taken as a part of the audio classifying methoddescribed here, and the detail is omitted. Briefly, the method mayfurther comprise measuring the lasting time during which the operationof identifying the context type continuously outputs the same contexttype, wherein the audio classifying method is configured to continue tooutput the present context type until the length of the lasting time ofa new context type reaches a sixth threshold.

Similarly, different sixth thresholds may be set for differenttransition pairs from one context type to another context type. Inaddition, the sixth threshold may be negatively correlated with theconfidence value of the new context type.

As a modification to the transition scheme in the audio classifyingmethod specially directed to VoIP/non-VoIP classification, any one ormore of the first to fourth threshold for the present short-term segmentmay be set different depending on the context type of the lastshort-term segment.

Similar to the embodiments of the audio processing apparatus, anycombination of the embodiments of the audio processing method and theirvariations are practical on one hand; and on the other hand, everyaspect of the embodiments of the audio processing method and theirvariations may be separate solutions. In addition, any two or moresolutions described in this section may be combined with each other, andthese combinations may be further combined with any embodiment describedor implied in the other parts of this disclosure. Specifically, theaudio classifying method described here may be used in the audioprocessing method described before, especially the volume levelercontrolling method.

As discussed at the beginning of the Detailed Description of the presentapplication, the embodiment of the application may be embodied either inhardware or in software, or in both. FIG. 41 is a block diagramillustrating an exemplary system for implementing the aspects of thepresent application.

In FIG. 41, a central processing unit (CPU) 4201 performs variousprocesses in accordance with a program stored in a read only memory(ROM) 4202 or a program loaded from a storage section 4208 to a randomaccess memory (RAM) 4203. In the RAM 4203, data required when the CPU4201 performs the various processes or the like are also stored asrequired.

The CPU 4201, the ROM 4202 and the RAM 4203 are connected to one anothervia a bus 4204. An input/output interface 4205 is also connected to thebus 4204.

The following components are connected to the input/output interface4205: an input section 4206 including a keyboard, a mouse, or the like;an output section 4207 including a display such as a cathode ray tube(CRT), a liquid crystal display (LCD), or the like, and a loudspeaker orthe like; the storage section 4208 including a hard disk or the like;and a communication section 4209 including a network interface card suchas a LAN card, a modem, or the like. The communication section 4209performs a communication process via the network such as the internet.

A drive 4210 is also connected to the input/output interface 4205 asrequired. A removable medium 4211, such as a magnetic disk, an opticaldisk, a magneto-optical disk, a semiconductor memory, or the like, ismounted on the drive 4210 as required, so that a computer program readthere from is installed into the storage section 4208 as required.

In the case where the above-described components are implemented by thesoftware, the program that constitutes the software is installed fromthe network such as the internet or the storage medium such as theremovable medium 4211.

Please note the terminology used herein is for the purpose of describingparticular embodiments only and is not intended to be limiting of theapplication. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, operations, steps,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, operations, steps,elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or operation plus function elements in the claims below areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present application has been presentedfor purposes of illustration and description, but is not intended to beexhaustive or limited to the application in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of theapplication. The embodiment was chosen and described in order to bestexplain the principles of the application and the practical application,and to enable others of ordinary skill in the art to understand theapplication for various embodiments with various modifications as aresuited to the particular use contemplated.

We claim:
 1. A loudness normalization method based upon a targetloudness value, the method comprising: determining dynamic gainparameters to be applied to audio segments of an audio signal based uponshort-term or long-term characteristics of the audio signal, saiddetermining including determining one or more dynamic gain parametersfor a first audio segment based on a short-term characteristic of theaudio signal and the target loudness value, and determining one or moredynamic gain parameters for a second audio segment based on a long-termcharacteristic of the audio signal and the target loudness value;modifying a loudness of the first audio segment by employing the one ormore dynamic gain parameters for the first audio segment; and modifyinga loudness of the second audio segment by employing the one or moredynamic gain parameters for the second audio segment, wherein thelong-term characteristic is determined differently from the short-termcharacteristic, and a loudness of the audio signal on playback isconsistent with the target loudness value over the audio segments. 2.The loudness normalization method of claim 1, wherein the dynamic gainparameters are identified and applied in real time.
 3. The loudnessnormalization method of claim 1, wherein dialog enhancement is appliedhaving an effect of making dialog more prominent within a particularcontext.
 4. The loudness normalization method of claim 1, wherein aloudness equalization is applied to have an effect on one or moreplayback levels on a tonal balance.
 5. The loudness normalization methodof claim 1, wherein a parameter smoothing is applied to the dynamic gainparameters.
 6. An audio processing apparatus configured to normalizeloudness based upon a target loudness value, comprising: at least oneprocessor; and at least one memory storing a computer program; in whichthe at least one memory with the computer program is configured with theat least one processor to cause the audio processing apparatus to atleast: determine dynamic gain parameters to be applied audio segments ofan audio signal based upon short-term or long-term characteristics ofthe audio signal, said determining including determining one or moredynamic gain parameters for a first audio segment based on a short-termcharacteristic of the audio signal and the target loudness value, anddetermining one or more dynamic gain parameters for a second audiosegment based on a long-term characteristic of the audio signal and thetarget loudness value; modify a loudness of the first audio segment byemploying the one or more dynamic gain parameters for the first audiosegment; and modify a loudness of the second audio segment by employingthe one or more dynamic gain parameters for the second audio segment,wherein the long-term characteristic is determined differently from theshort-term characteristic, and a loudness of the audio signal onplayback is consistent with the target loudness value over the audiosegments.
 7. The apparatus of claim 6, wherein the dynamic gainparameters are identified and applied in real time.
 8. The apparatus ofclaim 6, wherein dialog enhancement is applied having an effect ofmaking dialog more prominent within a particular context.
 9. Theapparatus of claim 6, wherein a loudness equalization is applied to havean effect on one or more playback levels on a tonal balance.
 10. Theapparatus of claim 6, wherein a parameter smoothing is applied to thedynamic gain parameters.
 11. A program storage device readable by amachine, tangibly embodying a program of instructions executable by themachine for causing to perform operations, said operations comprising:determining dynamic gain parameters to be applied to audio segments ofan audio signal based upon short-term or long-term characteristics ofthe audio signal, said determining including determining one or moredynamic gain parameters for a first audio segment based on a short-termcharacteristic of the audio signal and the target loudness value, anddetermining one or more dynamic gain parameters for a second audiosegment based on a long-term characteristic of the audio signal and thetarget loudness value; modifying a loudness of the first audio segmentby employing the one or more dynamic gain parameters for the first audiosegment; and modifying a loudness of the second audio segment byemploying the one or more dynamic gain parameters for the second audiosegment, wherein the long-term characteristic is determined differentlyfrom the short-term characteristic, and a loudness of the audio signalon playback is consistent with the target loudness value over the audiosegments.